2024-03-25T17:51:28.509 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:51:28.509 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/orig.config.yaml 2024-03-25T17:51:28.550 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:51:28.638 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621314' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA2kYWyeLWstO2L6gPtqoaupxwqZeJGQBC0Y1ga9le3tLy/Ar4QYnSk9MoRJYwdqZ0bE0cpZ8jOK54Ch+/4cOEA= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGX+Tafmgyf1WOPecYLO/4QNjiuEFjSTSXj4gKaNTmmczFBACEc92+GuTsy4TP3mDPyVmz5aT9XP/oSlYbNjtE= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:51:28.638 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:51:28.639 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:51:28.639 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:51:28.640 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:51:28.663 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:51:28.663 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:51:28.663 INFO:teuthology.packaging:ref: None 2024-03-25T17:51:28.663 INFO:teuthology.packaging:tag: None 2024-03-25T17:51:28.663 INFO:teuthology.packaging:branch: main 2024-03-25T17:51:28.663 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:51:28.663 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:51:28.880 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:51:28.881 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:51:28.888 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:51:28.888 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:51:28.903 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:51:28.917 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:51:28.926 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:51:28.950 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.417185', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA2kYWyeLWstO2L6gPtqoaupxwqZeJGQBC0Y1ga9le3tLy/Ar4QYnSk9MoRJYwdqZ0bE0cpZ8jOK54Ch+/4cOEA='} 2024-03-25T17:51:28.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.415660', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGX+Tafmgyf1WOPecYLO/4QNjiuEFjSTSXj4gKaNTmmczFBACEc92+GuTsy4TP3mDPyVmz5aT9XP/oSlYbNjtE='} 2024-03-25T17:51:28.970 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:51:28.975 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:51:28.976 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:51:28.976 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:51:29.043 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f218b9740d0>, signals=[15]) 2024-03-25T17:51:29.043 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:51:29.051 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:51:29.051 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:51:29.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:29.146 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:51:29.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:29.235 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:51:29.243 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-03-25T17:51:29.260 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-03-25T17:51:29.260 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-03-25T17:51:29.316 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:29.317 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:29.318 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-03-25T17:51:29.337 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-03-25T17:51:29.355 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-03-25T17:51:29.355 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:29.411 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:29.412 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:29.412 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:29.412 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:29.412 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:29.412 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:29.412 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-03-25T17:51:29.437 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:51:29.448 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:51:29.456 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:51:29.456 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:29.458 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:29.475 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:51:29.485 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:51:29.486 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:29.515 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:29.622 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:51:29.677 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:51:29.693 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:51:29.694 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:51:29.694 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:29.695 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:29.711 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-03-25T17:51:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:29.712 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-03-25T17:51:29.766 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:29.767 INFO:teuthology.task.kernel:Running kernel on smithi159: 5.14.0-361.el9.x86_64 2024-03-25T17:51:29.767 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2024-03-25T17:51:29.768 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:29.768 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-03-25T17:51:29.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-03-25T17:51:31.944 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T17:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T17:51:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-25T17:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-25T17:51:41.193 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-25T17:51:41.340 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-25T17:51:49.980 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-03-25T17:51:50.344 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-03-25T17:52:00.499 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 888 kB/s | 6.1 MB 00:07 2024-03-25T17:52:02.345 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-25T17:52:05.717 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-03-25T17:52:10.546 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 377 kB/s | 6.1 MB 00:16 2024-03-25T17:52:10.613 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-25T17:52:12.728 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:52:12.729 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:52:12.820 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:52:12.829 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:52:12.829 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:12.829 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:52:12.829 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:52:12.829 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:52:12.829 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T17:52:12.829 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:52:12.830 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 76 M 2024-03-25T17:52:12.831 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 123 M 2024-03-25T17:52:12.831 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:52:12.903 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:12.903 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:12.903 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:12.994 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:12.994 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:13.002 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:13.018 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T17:52:13.473 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:13.495 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:13.520 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:13.791 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:13.813 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:13.841 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:13.858 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:13.883 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:13.940 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:13.999 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:14.007 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:14.039 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:14.048 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:14.107 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:14.127 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:14.183 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:14.194 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:14.204 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:14.211 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:14.214 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:14.273 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:14.309 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:14.309 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:14.346 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:14.375 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:14.375 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:14.454 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:14.512 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:14.545 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:14.592 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:14.675 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:14.676 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:14.694 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:14.694 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:14.712 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:14.730 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:14.730 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:14.730 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:14.826 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:14.843 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:14.844 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:14.862 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:15.004 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:15.006 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:15.023 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T17:52:15.352 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:15.383 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:15.419 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:15.421 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:15.443 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:15.491 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:15.539 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:15.562 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:15.602 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:15.705 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:15.742 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:15.784 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:15.819 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:15.852 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:15.870 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:15.870 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:15.887 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:15.946 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:15.962 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:16.084 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:16.119 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:16.214 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 20 kB/s | 66 kB 00:03 2024-03-25T17:52:16.267 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:16.284 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:16.564 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.1 MB/s | 15 MB 00:03 2024-03-25T17:52:16.717 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:16.776 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:16.859 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:16.911 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:16.928 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:16.990 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:17.043 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:17.283 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:17.520 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:18.056 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.5 MB/s | 33 MB 00:05 2024-03-25T17:52:18.079 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:18.122 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:18.195 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:18.215 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:18.341 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:18.405 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:18.491 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:18.549 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:18.585 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:18.621 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:18.783 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:18.872 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:18.930 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:19.281 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:19.433 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:19.546 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:19.609 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:19.686 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:19.761 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:20.052 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-25T17:52:20.731 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.1 MB/s | 27 MB 00:04 2024-03-25T17:52:20.736 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:20.736 INFO:teuthology.orchestra.run.smithi134.stdout:Total 9.6 MB/s | 76 MB 00:07 2024-03-25T17:52:21.393 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:52:21.743 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:52:21.744 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:52:22.135 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:52:22.135 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:52:22.228 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:52:22.234 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:52:22.234 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:22.234 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:52:22.234 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:52:22.234 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout:Install 4 Packages 2024-03-25T17:52:22.235 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:52:22.236 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 76 M 2024-03-25T17:52:22.236 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 123 M 2024-03-25T17:52:22.236 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:52:22.372 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:22.372 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:22.372 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:22.463 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:22.464 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:22.470 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:22.625 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:22.626 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:22.647 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:22.693 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:22.693 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:22.714 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:22.783 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:52:22.783 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:52:22.836 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:22.849 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:22.873 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:23.304 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:23.325 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:23.361 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:23.626 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:23.650 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:23.667 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:23.712 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:23.713 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:23.756 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:23.757 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:23.778 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:23.798 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:23.822 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:23.842 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:23.902 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:23.963 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:23.999 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:24.007 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:24.024 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:24.039 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:24.056 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:24.160 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:24.192 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:24.228 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:24.519 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:24.542 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:24.612 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:24.679 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:24.704 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:24.791 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:24.792 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:24.820 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:24.926 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:24.948 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:24.969 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:24.993 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:25.027 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:25.064 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:25.064 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.082 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.083 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.083 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:25.116 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.125 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:25.162 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.162 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:25.198 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.214 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:25.218 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:25.242 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:52:25.254 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:25.290 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:25.291 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:25.328 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:25.329 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:25.456 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:25.469 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:25.498 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:25.499 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:25.509 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:25.563 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:25.583 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:25.583 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:25.641 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:25.695 INFO:teuthology.orchestra.run.smithi159.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 20 kB/s | 66 kB 00:03 2024-03-25T17:52:25.749 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.766 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:25.855 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:25.962 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.137 INFO:teuthology.orchestra.run.smithi159.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.9 MB/s | 15 MB 00:03 2024-03-25T17:52:26.162 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.290 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.357 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.854 INFO:teuthology.orchestra.run.smithi159.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.4 MB/s | 33 MB 00:04 2024-03-25T17:52:26.931 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:27.400 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:27.489 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:27.529 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:27.653 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.720 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.881 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:27.916 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:28.057 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:28.418 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:28.536 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:28.625 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:28.679 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:28.696 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:28.720 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:28.737 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:28.779 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:28.813 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:28.886 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:28.917 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:28.919 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:29.704 INFO:teuthology.orchestra.run.smithi159.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.9 MB/s | 27 MB 00:04 2024-03-25T17:52:29.709 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:29.709 INFO:teuthology.orchestra.run.smithi159.stdout:Total 10 MB/s | 76 MB 00:07 2024-03-25T17:52:30.373 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:52:30.784 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:52:30.784 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:52:31.789 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:52:31.789 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:52:33.635 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:33.676 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:34.280 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:52:34.980 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:35.071 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:37.817 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:37.958 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:38.390 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:47.488 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:07.549 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:07.549 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:53:07.549 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:53:07.593 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:07.942 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:07.942 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:53:07.942 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:53:07.942 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:53:08.383 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:53:08.733 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:08.836 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:08.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:08.837 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:53:08.837 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:53:08.837 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:53:08.837 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:53:08.837 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:53:08.837 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:08.917 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:08.917 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:08.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:53:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:53:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:53:08.964 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:53:08.964 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:53:08.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:53:09.576 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-03-25T17:53:11.044 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:53:11.080 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-03-25T17:53:11.083 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-25T17:53:11.098 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.89rvQKZwme 2024-03-25T17:53:11.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.89rvQKZwme 2024-03-25T17:53:11.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.89rvQKZwme 2024-03-25T17:53:11.317 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.89rvQKZwme is 6KB 2024-03-25T17:53:11.365 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.89rvQKZwme 2024-03-25T17:53:11.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:53:11.444 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:53:11.444 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:53:11.444 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:53:11.444 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:53:11.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:53:11.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-03-25T17:53:16.872 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:16.872 INFO:teuthology.orchestra.run.smithi159.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:53:16.872 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:53:16.918 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:17.300 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:17.300 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:53:17.300 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:53:17.300 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:53:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:53:17.718 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:53:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:53:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:53:17.719 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:53:18.050 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:18.191 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:18.191 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:18.191 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:53:18.191 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:53:18.191 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:53:18.191 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:53:18.191 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:53:18.191 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:18.271 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:18.272 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:18.272 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2024-03-25T17:53:18.306 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2024-03-25T17:53:18.306 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2024-03-25T17:53:18.306 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.06 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 68.el9 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Group : Unspecified 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 8264254 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : builder@centos.org 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : CentOS 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2024-03-25T17:53:18.307 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2024-03-25T17:53:18.308 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:53:18.308 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:53:18.308 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:53:18.308 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:53:18.308 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2024-03-25T17:53:18.308 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:53:18.308 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:53:18.309 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:53:18.309 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:53:18.309 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:53:18.926 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2024-03-25T17:53:20.304 INFO:teuthology.orchestra.run.smithi159.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:53:20.349 INFO:teuthology.orchestra.run.smithi159.stderr:done 2024-03-25T17:53:20.352 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-25T17:53:20.367 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.k97P76YMcu 2024-03-25T17:53:20.367 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.k97P76YMcu 2024-03-25T17:53:20.435 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.k97P76YMcu 2024-03-25T17:53:20.571 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.k97P76YMcu is 6KB 2024-03-25T17:53:20.620 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.k97P76YMcu 2024-03-25T17:53:20.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:53:20.700 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:53:20.700 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:53:20.700 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:53:20.700 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:53:20.701 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:53:20.848 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2024-03-25T17:53:41.631 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:53:41.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T17:53:41.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:50.859 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:53:50.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T17:53:50.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:25.439 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-03-25T17:54:34.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T17:54:34.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:41.640 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:54:50.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T17:54:50.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:51.097 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-03-25T17:54:51.588 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T17:54:51.588 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:54:51.589 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-03-25T17:54:51.603 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:54:51.603 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:54:51.603 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:54:51.604 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:54:52.604 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:54:52.605 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:54:52.675 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-03-25T17:54:52.722 DEBUG:teuthology.parallel:result is None 2024-03-25T17:55:34.451 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:55:46.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T17:55:46.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:55:46.927 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-03-25T17:55:47.335 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T17:55:47.336 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:55:47.336 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-03-25T17:55:47.351 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:55:47.351 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:55:47.351 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:55:47.351 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:55:48.352 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:55:48.352 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:55:48.431 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2024-03-25T17:55:48.465 DEBUG:teuthology.parallel:result is None 2024-03-25T17:55:48.466 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:55:48.475 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:55:48.475 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:55:48.478 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:55:48.495 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:55:48.502 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:55:48.515 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:55:48.516 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:55:48.537 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:55:48.684 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:55:48.691 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:55:48.691 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:55:48.694 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:55:48.726 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:48.740 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:48.743 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:48.745 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:55:48.756 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:55:48.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:55:48.787 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:55:48.816 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:55:48.832 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:55:48.832 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:55:48.859 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:55:48.875 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:48.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:49.043 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:49.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:49.129 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:55:49.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:55:49.196 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:49.233 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:49.294 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:49.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:49.421 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:55:49.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:55:49.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-03-25T17:55:49.491 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-03-25T17:55:49.533 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:55:49.568 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:55:49.927 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:55:49.939 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:55:49.939 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:55:49.956 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:55:49.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-03-25T17:55:50.018 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-03-25T17:55:50.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-03-25T17:55:50.077 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-03-25T17:55:50.079 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:55:50.079 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-03-25T17:55:50.097 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-03-25T17:55:50.097 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-03-25T17:55:50.132 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-03-25T17:55:50.132 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:55:50.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:55:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:55:50.167 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:55:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:55:50.200 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:55:50.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:55:50.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:55:50.278 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:55:50.289 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:55:50.306 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:55:50.308 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:00:17.544 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-03-25T18:00:17.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T18:00:17.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:00:17.636 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-03-25T18:00:17.716 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T18:00:17.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T18:00:17.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:00:17.806 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-03-25T18:00:17.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T18:00:17.887 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:00:17.898 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:00:17.899 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:00:17.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:00:17.901 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:00:17.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:00:17.930 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:00:17.947 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:00:17.959 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:00:17.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-03-25T18:00:17.976 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:00:17.989 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-03-25T18:00:18.006 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:00:18.006 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-03-25T18:00:18.020 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-03-25T18:00:18.022 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:00:18.036 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:00:18.053 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:00:18.073 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:00:18.076 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:00:18.076 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-25T18:00:18.077 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.077 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.077 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.077 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.106 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:00:18.109 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:00:18.109 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-25T18:00:18.110 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.110 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.110 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.110 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:00:18.111 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:00:18.119 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:00:18.119 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:00:18.119 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:00:18.119 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:00:18.125 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:00:18.125 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:00:18.125 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:00:18.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:00:18.127 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:00:18.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:00:18.267 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:00:18.267 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:00:18.359 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:00:18.359 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:00:18.599 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:00:18.600 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T18:00:18.600 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:00:18.630 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:00:18.630 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:00:18.697 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:00:18.697 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T18:00:18.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:00:18.700 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:00:18.726 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:00:18.726 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:00:18.781 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:00:18.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:00:18.849 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-03-25T18:00:18.851 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-03-25T18:00:18.876 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:00:18.942 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-03-25T18:00:18.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-03-25T18:00:19.105 INFO:teuthology.orchestra.run.smithi159.stdout:74 files removed 2024-03-25T18:00:19.133 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw 2024-03-25T18:00:19.190 INFO:teuthology.orchestra.run.smithi134.stdout:74 files removed 2024-03-25T18:00:19.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-03-25T18:00:20.689 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 65 kB/s | 82 kB 00:01 2024-03-25T18:00:20.726 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 68 kB/s | 82 kB 00:01 2024-03-25T18:00:21.622 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T18:00:21.701 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T18:00:22.526 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T18:00:22.556 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-25T18:00:22.953 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T18:00:23.042 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T18:00:24.654 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-25T18:00:24.709 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-25T18:00:33.603 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 956 kB/s | 8.0 MB 00:08 2024-03-25T18:00:34.240 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 896 kB/s | 8.0 MB 00:09 2024-03-25T18:00:39.137 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 19 MB 00:03 2024-03-25T18:00:40.446 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 19 MB 00:05 2024-03-25T18:00:51.291 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 730 kB/s | 6.1 MB 00:08 2024-03-25T18:00:52.351 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 751 kB/s | 6.1 MB 00:08 2024-03-25T18:00:52.920 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T18:00:54.081 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-25T18:00:54.908 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T18:00:55.918 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T18:00:59.830 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-25T18:01:00.857 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T18:01:02.298 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:01:02.313 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:01:02.314 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:01:02.315 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:01:02.316 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout:Installing weak dependencies: 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout:Install 30 Packages 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 2 Packages 2024-03-25T18:01:02.317 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:01:02.318 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 58 M 2024-03-25T18:01:02.318 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:01:03.318 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:01:03.328 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:01:03.328 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:01:03.328 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:01:03.328 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:01:03.328 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:01:03.328 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:01:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout:Install 30 Packages 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-03-25T18:01:03.331 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:01:03.332 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 58 M 2024-03-25T18:01:03.332 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:01:20.040 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 339 kB/s | 5.3 MB 00:16 2024-03-25T18:01:20.482 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 57 kB/s | 25 kB 00:00 2024-03-25T18:01:22.915 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 305 kB/s | 742 kB 00:02 2024-03-25T18:01:24.438 INFO:teuthology.orchestra.run.smithi159.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 252 kB/s | 5.3 MB 00:21 2024-03-25T18:01:24.655 INFO:teuthology.orchestra.run.smithi159.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 116 kB/s | 25 kB 00:00 2024-03-25T18:01:25.057 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 233 kB/s | 498 kB 00:02 2024-03-25T18:01:26.930 INFO:teuthology.orchestra.run.smithi159.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 326 kB/s | 742 kB 00:02 2024-03-25T18:01:28.488 INFO:teuthology.orchestra.run.smithi159.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 320 kB/s | 498 kB 00:01 2024-03-25T18:01:30.621 INFO:teuthology.orchestra.run.smithi159.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 358 kB/s | 9.7 MB 00:27 2024-03-25T18:01:30.838 INFO:teuthology.orchestra.run.smithi159.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 208 kB/s | 45 kB 00:00 2024-03-25T18:01:31.430 INFO:teuthology.orchestra.run.smithi159.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 222 kB/s | 131 kB 00:00 2024-03-25T18:01:33.522 INFO:teuthology.orchestra.run.smithi159.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 77 kB/s | 162 kB 00:02 2024-03-25T18:01:35.430 INFO:teuthology.orchestra.run.smithi159.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 168 kB/s | 320 kB 00:01 2024-03-25T18:01:36.622 INFO:teuthology.orchestra.run.smithi159.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 250 kB/s | 298 kB 00:01 2024-03-25T18:01:37.139 INFO:teuthology.orchestra.run.smithi159.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 192 kB/s | 99 kB 00:00 2024-03-25T18:01:37.247 INFO:teuthology.orchestra.run.smithi159.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T18:01:37.298 INFO:teuthology.orchestra.run.smithi159.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:01:37.331 INFO:teuthology.orchestra.run.smithi159.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:01:37.390 INFO:teuthology.orchestra.run.smithi159.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:01:37.449 INFO:teuthology.orchestra.run.smithi159.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-25T18:01:37.491 INFO:teuthology.orchestra.run.smithi159.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2024-03-25T18:01:37.570 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:37.661 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:37.680 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:37.753 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:37.913 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:38.265 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:38.301 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:38.342 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:38.433 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:38.498 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:38.628 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:38.749 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:38.766 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:38.835 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:38.968 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:39.089 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 283 kB/s | 9.7 MB 00:35 2024-03-25T18:01:39.297 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 217 kB/s | 45 kB 00:00 2024-03-25T18:01:39.454 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:39.748 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 292 kB/s | 131 kB 00:00 2024-03-25T18:01:39.781 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:39.941 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:40.240 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 329 kB/s | 162 kB 00:00 2024-03-25T18:01:40.295 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:40.372 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:40.512 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:40.569 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:40.586 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:40.748 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:40.837 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:40.965 INFO:teuthology.orchestra.run.smithi159.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 64 kB/s | 221 kB 00:03 2024-03-25T18:01:41.011 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:41.100 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:41.118 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:41.171 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:41.333 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:41.581 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 239 kB/s | 320 kB 00:01 2024-03-25T18:01:41.688 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:41.724 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:41.770 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:41.863 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:41.927 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:42.052 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:42.177 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:42.197 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:42.264 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:42.405 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:42.877 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:43.213 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:43.374 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:43.607 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 147 kB/s | 298 kB 00:02 2024-03-25T18:01:43.725 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:43.800 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:43.962 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:44.017 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:44.034 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:44.177 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:44.265 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:44.340 INFO:teuthology.orchestra.run.smithi159.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 11 kB/s | 36 kB 00:03 2024-03-25T18:01:44.383 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:44.472 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:44.491 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:44.583 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:44.590 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 101 kB/s | 99 kB 00:00 2024-03-25T18:01:44.699 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T18:01:44.744 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:44.749 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:01:44.783 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:01:44.841 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:01:44.909 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 263 kB/s | 5.1 MB 00:19 2024-03-25T18:01:44.934 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-25T18:01:44.959 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 831 kB/s | 41 kB 00:00 2024-03-25T18:01:45.023 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:45.023 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:45.097 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:45.132 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:45.148 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:45.173 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:45.178 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:45.225 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:45.249 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:45.278 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:45.342 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:45.368 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:45.394 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:45.467 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:45.531 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:45.556 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:45.588 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:45.606 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:45.672 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:45.805 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:45.883 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:45.906 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:45.920 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:45.941 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:45.962 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:45.981 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:46.057 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:46.078 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:46.122 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:46.145 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:46.254 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:46.277 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:46.278 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:46.401 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:46.425 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:46.489 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:46.500 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:46.536 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:46.546 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:46.617 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:46.630 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:46.648 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:46.777 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:47.100 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:47.125 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:47.127 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:47.203 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:47.343 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:47.401 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:47.418 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:47.426 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:47.443 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:47.445 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:47.484 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:47.517 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:47.536 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:47.560 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:47.637 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:47.647 INFO:teuthology.orchestra.run.smithi159.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:47.654 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:47.657 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:47.674 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:47.740 INFO:teuthology.orchestra.run.smithi159.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 14 kB/s | 47 kB 00:03 2024-03-25T18:01:47.814 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:47.833 INFO:teuthology.orchestra.run.smithi159.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.3 MB/s | 309 kB 00:00 2024-03-25T18:01:47.838 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:47.966 INFO:teuthology.orchestra.run.smithi159.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-03-25T18:01:47.999 INFO:teuthology.orchestra.run.smithi159.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 784 kB/s | 26 kB 00:00 2024-03-25T18:01:48.033 INFO:teuthology.orchestra.run.smithi159.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-25T18:01:48.066 INFO:teuthology.orchestra.run.smithi159.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-25T18:01:48.108 INFO:teuthology.orchestra.run.smithi159.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 20 MB/s | 838 kB 00:00 2024-03-25T18:01:48.142 INFO:teuthology.orchestra.run.smithi159.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-03-25T18:01:48.162 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:48.182 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:48.201 INFO:teuthology.orchestra.run.smithi159.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-25T18:01:48.300 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 11 kB/s | 36 kB 00:03 2024-03-25T18:01:48.326 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 65 kB/s | 221 kB 00:03 2024-03-25T18:01:48.365 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:48.487 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:48.562 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:48.701 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:01:48.863 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:49.159 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 371 kB/s | 309 kB 00:00 2024-03-25T18:01:49.239 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:49.274 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:49.315 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:49.431 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:49.495 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:49.620 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:49.758 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:49.851 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:01:49.897 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:49.989 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:50.457 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:50.796 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:50.813 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:50.864 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:50.983 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:51.002 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:51.160 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:51.506 INFO:teuthology.orchestra.run.smithi134.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:51.601 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 14 kB/s | 47 kB 00:03 2024-03-25T18:01:52.068 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 56 kB/s | 26 kB 00:00 2024-03-25T18:01:52.251 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-03-25T18:01:52.485 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 289 kB/s | 67 kB 00:00 2024-03-25T18:01:52.752 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 417 kB/s | 20 MB 00:48 2024-03-25T18:01:53.477 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 264 kB/s | 192 kB 00:00 2024-03-25T18:01:53.677 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 703 kB/s | 838 kB 00:01 2024-03-25T18:01:54.309 INFO:teuthology.orchestra.run.smithi159.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 202 kB/s | 5.1 MB 00:25 2024-03-25T18:01:54.719 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 814 kB/s | 4.4 MB 00:05 2024-03-25T18:01:55.553 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 780 kB/s | 1.6 MB 00:02 2024-03-25T18:02:01.294 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 454 kB/s | 3.4 MB 00:07 2024-03-25T18:02:04.033 INFO:teuthology.orchestra.run.smithi159.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 218 kB/s | 3.4 MB 00:15 2024-03-25T18:02:09.116 INFO:teuthology.orchestra.run.smithi159.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 216 kB/s | 3.1 MB 00:14 2024-03-25T18:02:10.084 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 208 kB/s | 3.1 MB 00:15 2024-03-25T18:02:10.094 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:10.094 INFO:teuthology.orchestra.run.smithi134.stdout:Total 887 kB/s | 58 MB 01:06 2024-03-25T18:02:10.126 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:02:10.165 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:02:10.165 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:02:10.670 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:02:10.670 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:02:11.766 INFO:teuthology.orchestra.run.smithi159.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 295 kB/s | 20 MB 01:09 2024-03-25T18:02:11.771 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:11.771 INFO:teuthology.orchestra.run.smithi159.stdout:Total 853 kB/s | 58 MB 01:09 2024-03-25T18:02:11.802 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:02:11.840 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:02:11.840 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:02:12.116 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:02:12.197 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:02:12.231 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:02:12.257 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:02:12.288 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:12.302 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:02:12.302 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:02:12.304 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:12.535 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:12.939 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:02:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:13.139 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:02:13.168 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:13.231 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:13.260 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:02:13.304 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:02:13.372 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:13.395 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:02:13.604 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:02:13.645 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:02:13.686 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:02:13.719 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:02:13.748 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:02:13.780 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:02:13.794 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:02:13.882 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:02:13.920 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:02:13.945 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:02:13.968 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:13.985 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:14.095 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:02:14.119 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:14.147 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:14.192 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:14.199 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:14.224 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:02:14.273 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:02:14.524 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:02:14.614 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:14.706 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:02:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:14.796 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:02:14.826 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:14.832 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:14.893 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:02:14.895 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:14.924 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:02:14.924 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:02:14.968 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:02:15.036 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:15.059 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:15.123 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:15.158 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:02:15.266 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:02:15.309 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:02:15.350 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:02:15.382 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:02:15.402 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:02:15.436 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:02:15.763 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:02:15.791 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:15.880 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:15.912 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:02:15.952 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:02:16.400 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:02:16.426 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:02:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:02:17.092 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:17.137 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:17.695 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:17.743 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:17.838 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:17.838 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:02:17.838 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:02:17.838 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:02:17.878 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:18.783 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:18.825 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:19.366 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:19.491 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:19.492 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:02:19.492 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:02:19.492 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:02:19.530 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:27.203 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-03-25T18:02:27.203 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-03-25T18:02:27.203 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-03-25T18:02:27.204 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-03-25T18:02:27.204 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-03-25T18:02:27.204 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-03-25T18:02:27.204 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-03-25T18:02:27.204 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:02:28.116 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:28.140 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:28.140 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:02:28.140 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:02:28.140 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:28.140 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:28.140 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:02:28.142 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:28.235 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:28.235 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:28.845 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /root 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-03-25T18:02:28.846 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:02:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:02:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:02:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:02:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:02:29.720 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:29.744 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:29.744 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:02:29.744 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:02:29.745 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:29.745 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:29.745 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:02:29.746 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.565 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.566 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:02:30.567 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:02:30.568 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:02:30.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-03-25T18:02:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:02:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:02:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:02:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:02:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:02:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:02:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:02:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:02:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:02:31.398 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:31 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:02:31.549 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:02:31.549 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:02:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:02:31.551 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:02:31.551 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:02:31.551 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-03-25T18:02:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:02:31.551 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-03-25T18:02:31.551 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 204 M 2024-03-25T18:02:31.551 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:02:31.838 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:02:31.938 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.0 MB/s | 300 kB 00:00 2024-03-25T18:02:31.980 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:02:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:02:32.088 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-03-25T18:02:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.088 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:02:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:02:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:02:32.091 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:02:32.386 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test 2024-03-25T18:02:32.488 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 197 kB/s | 163 kB 00:00 2024-03-25T18:02:32.864 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 212 kB/s | 187 kB 00:00 2024-03-25T18:02:32.942 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:33 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:02:33.101 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:02:33.102 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:02:33.102 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:02:33.102 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:02:33.102 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2024-03-25T18:02:33.103 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:02:33.104 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 48 M 2024-03-25T18:02:33.104 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 204 M 2024-03-25T18:02:33.104 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:02:33.421 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:02:33.470 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-25T18:02:33.504 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:02:33.696 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 976 kB/s | 187 kB 00:00 2024-03-25T18:02:34.447 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 135 kB/s | 163 kB 00:01 2024-03-25T18:03:48.515 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 643 kB/s | 47 MB 01:15 2024-03-25T18:03:48.518 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:48.518 INFO:teuthology.orchestra.run.smithi159.stdout:Total 654 kB/s | 48 MB 01:15 2024-03-25T18:03:48.529 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:03:48.548 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:03:48.549 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:03:48.932 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:03:48.933 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:03:49.514 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:03:49.581 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:03:49.618 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:03:49.658 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:49.706 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:03:49.734 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:54.283 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:54.359 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:03:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:03:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:03:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:03:57.029 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:03:57.030 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:03:57.030 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:03:57.030 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:03:57.320 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph 2024-03-25T18:03:57.889 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:58 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:03:58.050 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:03:58.053 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-03-25T18:03:58.053 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:03:58.053 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:03:58.054 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:03:58.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout:Install 36 Packages 2024-03-25T18:03:58.056 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:03:58.057 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 30 M 2024-03-25T18:03:58.057 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 107 M 2024-03-25T18:03:58.057 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:03:59.026 INFO:teuthology.orchestra.run.smithi159.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T18:04:04.743 INFO:teuthology.orchestra.run.smithi159.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 250 kB/s | 1.5 MB 00:06 2024-03-25T18:04:07.143 INFO:teuthology.orchestra.run.smithi159.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 256 kB/s | 2.1 MB 00:08 2024-03-25T18:04:08.151 INFO:teuthology.orchestra.run.smithi159.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 245 kB/s | 247 kB 00:01 2024-03-25T18:04:08.368 INFO:teuthology.orchestra.run.smithi159.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 862 kB/s | 187 kB 00:00 2024-03-25T18:04:08.418 INFO:teuthology.orchestra.run.smithi159.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T18:04:08.451 INFO:teuthology.orchestra.run.smithi159.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 952 kB/s | 32 kB 00:00 2024-03-25T18:04:08.494 INFO:teuthology.orchestra.run.smithi159.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T18:04:08.536 INFO:teuthology.orchestra.run.smithi159.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 924 kB/s | 39 kB 00:00 2024-03-25T18:04:08.594 INFO:teuthology.orchestra.run.smithi159.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T18:04:08.670 INFO:teuthology.orchestra.run.smithi159.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:04:08.703 INFO:teuthology.orchestra.run.smithi159.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:04:08.737 INFO:teuthology.orchestra.run.smithi159.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:04:08.779 INFO:teuthology.orchestra.run.smithi159.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:04:08.971 INFO:teuthology.orchestra.run.smithi159.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 652 kB/s | 125 kB 00:00 2024-03-25T18:04:09.196 INFO:teuthology.orchestra.run.smithi159.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-25T18:04:09.638 INFO:teuthology.orchestra.run.smithi159.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 342 kB/s | 151 kB 00:00 2024-03-25T18:04:09.755 INFO:teuthology.orchestra.run.smithi159.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-03-25T18:04:09.939 INFO:teuthology.orchestra.run.smithi159.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 937 kB/s | 172 kB 00:00 2024-03-25T18:04:10.139 INFO:teuthology.orchestra.run.smithi159.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T18:04:10.239 INFO:teuthology.orchestra.run.smithi159.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T18:04:10.339 INFO:teuthology.orchestra.run.smithi159.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-25T18:04:10.448 INFO:teuthology.orchestra.run.smithi159.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 214 kB/s | 23 kB 00:00 2024-03-25T18:04:10.565 INFO:teuthology.orchestra.run.smithi159.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 167 kB/s | 19 kB 00:00 2024-03-25T18:04:10.665 INFO:teuthology.orchestra.run.smithi159.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 197 kB/s | 20 kB 00:00 2024-03-25T18:04:10.765 INFO:teuthology.orchestra.run.smithi159.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 464 kB/s | 46 kB 00:00 2024-03-25T18:04:10.866 INFO:teuthology.orchestra.run.smithi159.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 789 kB/s | 79 kB 00:00 2024-03-25T18:04:10.999 INFO:teuthology.orchestra.run.smithi159.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-25T18:04:11.108 INFO:teuthology.orchestra.run.smithi159.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-03-25T18:04:11.234 INFO:teuthology.orchestra.run.smithi159.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 718 kB/s | 90 kB 00:00 2024-03-25T18:04:11.367 INFO:teuthology.orchestra.run.smithi159.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 269 kB/s | 36 kB 00:00 2024-03-25T18:04:11.484 INFO:teuthology.orchestra.run.smithi159.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-25T18:04:11.626 INFO:teuthology.orchestra.run.smithi159.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-25T18:04:11.718 INFO:teuthology.orchestra.run.smithi159.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-25T18:04:13.342 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 476 kB/s | 47 MB 01:41 2024-03-25T18:04:13.344 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:13.345 INFO:teuthology.orchestra.run.smithi134.stdout:Total 485 kB/s | 48 MB 01:41 2024-03-25T18:04:13.356 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:04:13.375 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:04:13.375 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:04:13.752 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:04:13.753 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:04:14.347 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:04:14.426 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:04:14.462 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:04:14.503 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:04:14.547 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:04:14.570 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:04:15.059 INFO:teuthology.orchestra.run.smithi159.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 299 kB/s | 4.7 MB 00:16 2024-03-25T18:04:19.090 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:04:19.161 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:04:21.488 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:04:21.488 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:04:21.488 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:04:21.488 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:04:21.488 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:04:21.489 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:04:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:04:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:04:21.852 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:04:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:04:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:04:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:04:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:04:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:04:21.853 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:04:22.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-03-25T18:04:22.727 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:22 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:04:22.891 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:04:22.894 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-25T18:04:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:04:22.894 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-25T18:04:22.894 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:04:22.894 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:04:22.894 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:04:22.895 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:04:22.896 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:04:22.897 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-25T18:04:22.898 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-03-25T18:04:22.898 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:04:22.898 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-03-25T18:04:22.898 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 107 M 2024-03-25T18:04:22.898 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:04:24.153 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 7.2 kB/s | 6.6 kB 00:00 2024-03-25T18:04:29.502 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 347 kB/s | 2.1 MB 00:06 2024-03-25T18:04:29.819 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 231 kB/s | 1.5 MB 00:06 2024-03-25T18:04:30.786 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 256 kB/s | 247 kB 00:00 2024-03-25T18:04:30.903 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:04:30.945 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:04:30.978 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-25T18:04:31.012 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:04:31.045 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:04:31.087 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:04:31.179 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-25T18:04:31.213 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:04:31.263 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T18:04:31.297 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:04:31.439 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 881 kB/s | 125 kB 00:00 2024-03-25T18:04:31.564 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-25T18:04:31.790 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 671 kB/s | 151 kB 00:00 2024-03-25T18:04:31.848 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-03-25T18:04:31.948 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T18:04:32.057 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-03-25T18:04:32.116 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-25T18:04:32.174 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T18:04:32.233 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-03-25T18:04:32.291 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 334 kB/s | 19 kB 00:00 2024-03-25T18:04:32.350 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-03-25T18:04:32.409 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 794 kB/s | 46 kB 00:00 2024-03-25T18:04:32.467 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T18:04:32.543 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T18:04:32.601 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-25T18:04:32.660 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T18:04:32.719 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-03-25T18:04:32.786 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T18:04:32.861 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T18:04:32.919 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-25T18:04:42.735 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 258 kB/s | 4.7 MB 00:18 2024-03-25T18:05:00.354 INFO:teuthology.orchestra.run.smithi159.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 309 kB/s | 17 MB 00:55 2024-03-25T18:05:00.358 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:00.358 INFO:teuthology.orchestra.run.smithi159.stdout:Total 488 kB/s | 30 MB 01:02 2024-03-25T18:05:00.382 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:05:00.423 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:05:00.423 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:05:00.845 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:05:00.845 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:05:02.214 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:05:02.281 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:05:02.322 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:05:02.359 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:05:02.430 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:05:02.482 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:05:02.523 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:05:02.615 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:05:02.702 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:05:02.743 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:05:02.814 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:05:02.868 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:05:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:05:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:05:03.020 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:05:03.202 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:05:03.244 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:05:03.275 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:05:03.309 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:05:03.331 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:05:03.406 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:05:03.441 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:03.475 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:05:03.524 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:05:03.559 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:05:03.593 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:05:03.681 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:05:03.715 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:05:03.788 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:05:03.933 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:05:04.060 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:04.225 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:05:04.256 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:04.281 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:04.281 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:04.281 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:05:04.281 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:04.282 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:04.282 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:05.749 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:05.776 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:05.776 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:05.776 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:05:05.776 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:05.776 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:05.776 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:06.216 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:06.240 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:06.240 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:06.240 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:05:06.240 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:06.240 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:06.240 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:06.450 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:06.473 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:06.474 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:06.474 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:05:06.474 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:06.474 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:06.474 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:06.539 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:08.216 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:08.216 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:05:08.216 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:05:08.216 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:05:08.216 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:05:08.216 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:05:08.216 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:05:08.217 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:05:08.218 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:05:08.219 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:05:08.219 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:05:09.227 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:05:09.227 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:09.227 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:05:09.228 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:05:09.229 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:05:09.230 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:05:09.231 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:05:09.231 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:09.231 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:09.563 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base 2024-03-25T18:05:09.840 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 426 kB/s | 17 MB 00:40 2024-03-25T18:05:09.844 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:09.844 INFO:teuthology.orchestra.run.smithi134.stdout:Total 647 kB/s | 30 MB 00:46 2024-03-25T18:05:09.867 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:05:09.908 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:05:09.908 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:05:10.153 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:10.250 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:10.302 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:10.303 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:05:10.303 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:10.309 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:05:10.309 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:05:10.366 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install cephadm 2024-03-25T18:05:10.925 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:11.073 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:11.080 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 747 k 2024-03-25T18:05:11.081 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 754 k 2024-03-25T18:05:11.081 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:05:11.636 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:05:11.709 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:05:11.758 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:05:11.786 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:05:11.857 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:05:11.893 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:05:11.926 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:05:12.019 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:05:12.096 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:05:12.146 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:05:12.218 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:05:12.271 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:05:12.309 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:05:12.358 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:05:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:05:12.608 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:05:12.655 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:05:12.694 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:05:12.729 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:05:12.758 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:05:12.835 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:05:12.868 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:12.910 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:05:12.925 INFO:teuthology.orchestra.run.smithi159.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 404 kB/s | 747 kB 00:01 2024-03-25T18:05:12.925 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:12.926 INFO:teuthology.orchestra.run.smithi159.stdout:Total 404 kB/s | 747 kB 00:01 2024-03-25T18:05:12.926 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:05:12.932 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:05:12.932 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:05:12.938 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:05:12.939 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:05:12.951 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:05:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:05:13.015 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:05:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:05:13.133 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:05:13.182 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:05:13.255 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:05:13.392 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:05:13.494 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:13.579 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:13.682 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:05:13.720 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:13.746 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:13.746 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:13.746 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:05:13.746 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:13.746 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:13.746 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:13.871 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:14.166 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:14.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:05:14.167 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:14.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:14.380 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:05:14.954 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:15.108 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:15.109 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:05:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 145 k 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 436 k 2024-03-25T18:05:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:05:15.179 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:15.206 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:15.206 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:15.206 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:05:15.206 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:15.207 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:15.207 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:15.647 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:15.670 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:15.670 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:15.670 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:05:15.670 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:15.670 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:15.670 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:15.881 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:15.904 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:15.904 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:15.904 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:05:15.904 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:15.905 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:15.905 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:15.970 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:16.105 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 146 kB/s | 145 kB 00:00 2024-03-25T18:05:16.106 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:16.107 INFO:teuthology.orchestra.run.smithi159.stdout:Total 146 kB/s | 145 kB 00:00 2024-03-25T18:05:16.107 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:05:16.114 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:05:16.114 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:05:16.168 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:05:16.168 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:05:16.309 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:05:16.364 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:16.389 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:16.389 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:16.389 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:05:16.389 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:17.323 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:17.323 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:17.323 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:05:17.323 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:17.323 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:17.323 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:17.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:05:17.643 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:05:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:05:17.645 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:05:18.184 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:18.283 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:18.335 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:18.336 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:05:18.337 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:18.396 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:05:18.590 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:05:18.590 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:05:18.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:05:18.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:18.593 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:05:18.958 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:18.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-03-25T18:05:19.109 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:19.110 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-03-25T18:05:19.110 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:05:19.110 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-03-25T18:05:19.110 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:05:19.110 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:05:19.110 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:05:19.110 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-03-25T18:05:19.111 INFO:teuthology.orchestra.run.smithi159.stdout:Install 12 Packages 2024-03-25T18:05:19.112 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:19.112 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 6.9 M 2024-03-25T18:05:19.112 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 26 M 2024-03-25T18:05:19.112 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:05:19.557 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:05:19.655 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:19.708 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:05:19.709 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:05:19.709 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:05:19.767 INFO:teuthology.orchestra.run.smithi159.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 68 kB/s | 31 kB 00:00 2024-03-25T18:05:19.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-03-25T18:05:19.801 INFO:teuthology.orchestra.run.smithi159.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 31 kB/s | 15 kB 00:00 2024-03-25T18:05:19.943 INFO:teuthology.orchestra.run.smithi159.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-25T18:05:19.976 INFO:teuthology.orchestra.run.smithi159.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-25T18:05:20.118 INFO:teuthology.orchestra.run.smithi159.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-03-25T18:05:20.252 INFO:teuthology.orchestra.run.smithi159.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T18:05:20.277 INFO:teuthology.orchestra.run.smithi159.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-03-25T18:05:20.340 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:05:20.427 INFO:teuthology.orchestra.run.smithi159.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 964 kB/s | 144 kB 00:00 2024-03-25T18:05:20.486 INFO:teuthology.orchestra.run.smithi159.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.8 MB/s | 2.0 MB 00:00 2024-03-25T18:05:20.489 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:05:20.490 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:05:20.490 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:05:20.490 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:05:20.490 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:05:20.490 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:05:20.490 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:20.490 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:05:20.491 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:05:20.491 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:05:20.491 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:20.491 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 747 k 2024-03-25T18:05:20.491 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 754 k 2024-03-25T18:05:20.491 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:05:20.511 INFO:teuthology.orchestra.run.smithi159.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 373 kB/s | 31 kB 00:00 2024-03-25T18:05:20.553 INFO:teuthology.orchestra.run.smithi159.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T18:05:22.154 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 450 kB/s | 747 kB 00:01 2024-03-25T18:05:22.154 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:22.155 INFO:teuthology.orchestra.run.smithi134.stdout:Total 449 kB/s | 747 kB 00:01 2024-03-25T18:05:22.155 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:05:22.161 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:05:22.161 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:05:22.167 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:05:22.167 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:05:22.243 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:05:22.699 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:23.157 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:23.454 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:23.454 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:23.454 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:05:23.454 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:23.454 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:23.454 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:05:23.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:05:24.036 INFO:teuthology.orchestra.run.smithi159.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 386 kB/s | 1.8 MB 00:04 2024-03-25T18:05:24.042 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:24.042 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.4 MB/s | 6.9 MB 00:04 2024-03-25T18:05:24.061 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:05:24.080 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:05:24.080 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:05:24.242 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:05:24.263 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:05:24.264 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:05:24.395 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:05:24.396 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:05:24.396 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:05:24.396 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:05:24.396 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:05:24.396 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 145 k 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 436 k 2024-03-25T18:05:24.397 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:05:24.778 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:05:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:05:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:05:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:05:24.966 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:05:25.061 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:05:25.183 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:05:25.284 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:05:25.328 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:05:25.357 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:25.389 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:05:25.584 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 122 kB/s | 145 kB 00:01 2024-03-25T18:05:25.585 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:25.585 INFO:teuthology.orchestra.run.smithi134.stdout:Total 122 kB/s | 145 kB 00:01 2024-03-25T18:05:25.585 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:05:25.594 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:05:25.594 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:05:25.631 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:05:25.650 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:05:25.650 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:05:25.664 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:25.833 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:05:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:25.927 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:25.927 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:05:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:26.440 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:26.440 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:05:26.440 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:05:26.440 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:05:26.441 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:05:26.790 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:26.791 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:26.896 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:05:26.896 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:26.896 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:05:26.896 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:26.896 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:26.896 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:05:27.053 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:05:27.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-03-25T18:05:27.613 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:27.719 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:05:27.765 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:05:27.767 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-25T18:05:27.768 INFO:teuthology.orchestra.run.smithi159.stdout:Install 12 Packages 2024-03-25T18:05:27.769 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:27.769 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 37 M 2024-03-25T18:05:27.769 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 209 M 2024-03-25T18:05:27.769 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:05:27.818 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:27.870 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:05:27.871 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:05:27.871 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:05:27.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:05:28.193 INFO:teuthology.orchestra.run.smithi159.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 287 kB/s | 32 kB 00:00 2024-03-25T18:05:28.252 INFO:teuthology.orchestra.run.smithi159.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 489 kB/s | 17 kB 00:00 2024-03-25T18:05:28.285 INFO:teuthology.orchestra.run.smithi159.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:05:28.443 INFO:teuthology.orchestra.run.smithi159.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:05:28.504 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:05:28.577 INFO:teuthology.orchestra.run.smithi159.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:05:28.619 INFO:teuthology.orchestra.run.smithi159.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T18:05:28.654 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:05:28.655 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-25T18:05:28.655 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:05:28.655 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-25T18:05:28.655 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:05:28.655 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:05:28.655 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:28.656 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:05:28.657 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-25T18:05:28.657 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-03-25T18:05:28.657 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:28.657 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.9 M 2024-03-25T18:05:28.657 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-03-25T18:05:28.657 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:05:28.686 INFO:teuthology.orchestra.run.smithi159.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-03-25T18:05:28.987 INFO:teuthology.orchestra.run.smithi159.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T18:05:29.012 INFO:teuthology.orchestra.run.smithi159.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-25T18:05:29.087 INFO:teuthology.orchestra.run.smithi159.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-25T18:05:29.154 INFO:teuthology.orchestra.run.smithi159.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T18:05:29.420 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T18:05:29.529 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 50 kB/s | 31 kB 00:00 2024-03-25T18:05:29.579 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T18:05:29.613 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T18:05:29.713 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.3 MB/s | 863 kB 00:00 2024-03-25T18:05:29.955 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-25T18:05:30.230 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 888 kB/s | 548 kB 00:00 2024-03-25T18:05:30.347 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-25T18:05:30.439 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 336 kB/s | 31 kB 00:00 2024-03-25T18:05:30.572 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-25T18:05:30.606 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-03-25T18:05:36.839 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 230 kB/s | 1.8 MB 00:07 2024-03-25T18:05:36.843 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:36.843 INFO:teuthology.orchestra.run.smithi134.stdout:Total 858 kB/s | 6.9 MB 00:08 2024-03-25T18:05:36.863 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:05:36.882 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:05:36.883 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:05:37.070 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:05:37.071 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:05:37.589 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:05:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:05:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:05:37.754 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:05:37.793 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:05:37.899 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:05:38.034 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:05:38.121 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:05:38.164 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:05:38.201 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:38.232 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:05:38.475 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:05:38.509 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:38.903 INFO:teuthology.orchestra.run.smithi159.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 699 kB/s | 7.4 MB 00:10 2024-03-25T18:05:38.906 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:38.906 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.3 MB/s | 37 MB 00:11 2024-03-25T18:05:38.951 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:05:38.976 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:05:38.976 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:05:39.338 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:05:39.338 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:05:39.350 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:39.350 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:05:39.350 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:05:39.350 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:05:39.350 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:05:39.351 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:05:39.351 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:05:39.351 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:05:39.351 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:05:39.351 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:39.351 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:05:39.351 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:05:39.699 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:05:39.699 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:39.699 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:05:39.699 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:05:39.699 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:39.699 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:39.700 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:05:39.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:05:40.037 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:05:40.100 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:05:40.132 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:05:40.225 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:40.260 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:05:40.437 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:40.467 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:40.517 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:05:40.551 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:05:40.668 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:05:40.669 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-25T18:05:40.669 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:05:40.669 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-25T18:05:40.669 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:05:40.669 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:05:40.669 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:05:40.670 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:40.671 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:05:40.671 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-25T18:05:40.671 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-03-25T18:05:40.671 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:05:40.671 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-03-25T18:05:40.671 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-03-25T18:05:40.672 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:05:41.193 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:05:41.212 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-03-25T18:05:41.245 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 520 kB/s | 17 kB 00:00 2024-03-25T18:05:41.303 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 586 kB/s | 34 kB 00:00 2024-03-25T18:05:41.304 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:05:41.495 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T18:05:41.746 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-25T18:05:41.787 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T18:05:41.971 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.3 MB/s | 3.0 MB 00:00 2024-03-25T18:05:42.163 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T18:05:42.289 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-25T18:05:42.313 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T18:05:42.397 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-25T18:05:42.551 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:05:43.575 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:05:43.600 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:46.156 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:46.156 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:05:46.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:05:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:05:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:05:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:05:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:05:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:46.633 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:47.097 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:05:47.671 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:47.822 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:47.823 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:05:47.823 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:05:47.823 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:05:47.823 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:05:47.823 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:05:47.824 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:05:47.825 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:47.825 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:05:47.825 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:05:47.825 INFO:teuthology.orchestra.run.smithi159.stdout:Install 13 Packages 2024-03-25T18:05:47.825 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:47.825 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 2.0 M 2024-03-25T18:05:47.826 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 26 M 2024-03-25T18:05:47.826 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:05:48.185 INFO:teuthology.orchestra.run.smithi159.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 130 kB/s | 24 kB 00:00 2024-03-25T18:05:48.201 INFO:teuthology.orchestra.run.smithi159.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 88 kB/s | 18 kB 00:00 2024-03-25T18:05:48.302 INFO:teuthology.orchestra.run.smithi159.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-03-25T18:05:48.327 INFO:teuthology.orchestra.run.smithi159.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-25T18:05:48.352 INFO:teuthology.orchestra.run.smithi159.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:05:48.378 INFO:teuthology.orchestra.run.smithi159.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-25T18:05:48.503 INFO:teuthology.orchestra.run.smithi159.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-25T18:05:48.528 INFO:teuthology.orchestra.run.smithi159.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-03-25T18:05:48.695 INFO:teuthology.orchestra.run.smithi159.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.0 MB/s | 203 kB 00:00 2024-03-25T18:05:48.754 INFO:teuthology.orchestra.run.smithi159.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-25T18:05:48.787 INFO:teuthology.orchestra.run.smithi159.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-25T18:05:48.837 INFO:teuthology.orchestra.run.smithi159.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-25T18:05:48.871 INFO:teuthology.orchestra.run.smithi159.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 60 kB/s | 52 kB 00:00 2024-03-25T18:05:48.873 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:48.873 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-25T18:05:48.874 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:05:48.889 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:05:48.889 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:05:48.987 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:05:48.988 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:05:49.524 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:05:49.693 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:05:49.748 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:05:49.803 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:05:49.840 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:05:49.881 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:05:49.984 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:05:50.093 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:05:50.149 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:05:50.878 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:05:50.930 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:05:50.964 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:05:51.012 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:05:51.033 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:51.773 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:05:51.774 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:05:52.127 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:05:52.127 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:52.127 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:05:52.127 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:05:52.128 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:52.129 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:52.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:05:52.946 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:53.096 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:53.097 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:05:53.097 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:05:53.097 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:05:53.097 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:05:53.097 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:05:53.097 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:05:53.097 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2024-03-25T18:05:53.098 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:53.099 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 6.8 M 2024-03-25T18:05:53.099 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 33 M 2024-03-25T18:05:53.099 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:05:53.607 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-25T18:05:53.706 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T18:05:53.773 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 878 kB/s | 58 kB 00:00 2024-03-25T18:05:53.816 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-25T18:05:53.841 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2024-03-25T18:05:54.391 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 145 kB/s | 138 kB 00:00 2024-03-25T18:05:54.393 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:54.393 INFO:teuthology.orchestra.run.smithi159.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2024-03-25T18:05:54.407 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:05:54.418 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:05:54.418 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:05:54.508 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:05:54.510 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:05:54.808 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:05:54.971 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:05:55.025 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:05:55.551 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:05:55.636 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:05:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:05:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:56.519 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:56.520 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:05:56.520 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:05:56.520 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:05:56.520 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:05:56.520 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:05:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:05:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:56.819 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:05:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:05:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:05:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:05:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:05:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:05:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:56.820 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:05:57.068 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse 2024-03-25T18:05:57.629 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:05:57.779 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:05:57.780 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:05:57.781 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-03-25T18:05:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:05:57.781 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 964 k 2024-03-25T18:05:57.781 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.8 M 2024-03-25T18:05:57.782 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:05:57.901 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 659 kB/s | 79 kB 00:00 2024-03-25T18:06:01.334 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 249 kB/s | 885 kB 00:03 2024-03-25T18:06:01.336 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi159.stdout:Total 271 kB/s | 964 kB 00:03 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:06:01.361 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:06:01.361 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:06:01.438 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:06:01.438 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:06:01.620 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:06:01.760 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:06:01.835 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:02.203 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 358 kB/s | 7.4 MB 00:21 2024-03-25T18:06:02.206 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:02.206 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.7 MB/s | 37 MB 00:21 2024-03-25T18:06:02.248 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:02.275 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:02.275 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:02.532 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:02.533 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:06:02.633 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:02.633 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:02.851 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:06:02.852 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:02.852 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:06:02.852 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:06:02.852 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:02.852 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:03.097 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-volume 2024-03-25T18:06:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:03.392 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:06:03.432 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:06:03.526 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:06:03.585 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:06:03.672 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:04 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:03.762 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:06:03.792 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:06:03.830 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:03.831 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:06:03.832 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:03.832 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-03-25T18:06:03.832 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:03.832 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 332 k 2024-03-25T18:06:03.832 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.5 M 2024-03-25T18:06:03.832 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:06:03.875 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:06:03.941 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 633 kB/s | 69 kB 00:00 2024-03-25T18:06:04.534 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:06:04.654 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:06:05.674 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 143 kB/s | 264 kB 00:01 2024-03-25T18:06:05.676 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:05.676 INFO:teuthology.orchestra.run.smithi159.stdout:Total 180 kB/s | 332 kB 00:01 2024-03-25T18:06:05.676 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:06:05.686 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:06:05.686 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:06:05.734 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:06:05.735 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:06:05.884 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:06:05.885 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:06:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:06:06.118 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:06.133 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:06.134 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:06.134 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:06:06.134 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:06.859 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:06:06.912 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:06:06.949 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:07.176 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:06:07.176 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:07.177 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:06:07.177 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:07.177 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:06:07.177 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:07.177 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:07.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel 2024-03-25T18:06:08.084 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:08.234 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 127 k 2024-03-25T18:06:08.235 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 456 k 2024-03-25T18:06:08.236 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:06:09.013 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 163 kB/s | 127 kB 00:00 2024-03-25T18:06:09.014 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:09.014 INFO:teuthology.orchestra.run.smithi159.stdout:Total 162 kB/s | 127 kB 00:00 2024-03-25T18:06:09.014 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:06:09.020 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:06:09.020 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:06:09.078 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:06:09.078 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:06:09.221 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:06:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:09.546 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:09.546 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:06:09.546 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:06:09.547 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:06:09.783 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:06:10.004 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:06:10.005 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:06:10.005 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:10.005 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:10.005 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:06:10.005 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:10.005 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:10.083 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:10.083 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:10.083 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:06:10.083 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:10.083 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:10.083 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:10.304 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2 2024-03-25T18:06:10.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:06:10.873 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:10.973 INFO:teuthology.orchestra.run.smithi159.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:10.993 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:11.029 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:11.030 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:06:11.030 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:11.096 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs-devel 2024-03-25T18:06:11.145 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:11.147 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:06:11.148 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-03-25T18:06:11.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:11.148 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-03-25T18:06:11.148 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-03-25T18:06:11.148 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:11.433 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 245 kB/s | 24 kB 00:00 2024-03-25T18:06:11.491 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-25T18:06:11.541 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-25T18:06:11.566 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:06:11.592 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T18:06:11.617 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-25T18:06:11.660 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:11.751 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-03-25T18:06:11.784 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-03-25T18:06:11.810 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:06:11.811 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:11.812 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:06:11.812 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:11.812 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 31 k 2024-03-25T18:06:11.812 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 140 k 2024-03-25T18:06:11.812 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:06:11.851 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.0 MB/s | 203 kB 00:00 2024-03-25T18:06:11.910 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 8.2 MB/s | 1.0 MB 00:00 2024-03-25T18:06:11.935 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 714 kB/s | 59 kB 00:00 2024-03-25T18:06:11.960 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-25T18:06:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 70 kB/s | 52 kB 00:00 2024-03-25T18:06:12.080 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:12.081 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T18:06:12.081 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:12.094 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:12.094 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:12.185 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:12.189 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:12.483 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 46 kB/s | 31 kB 00:00 2024-03-25T18:06:12.484 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:12.484 INFO:teuthology.orchestra.run.smithi159.stdout:Total 46 kB/s | 31 kB 00:00 2024-03-25T18:06:12.484 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:06:12.488 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:06:12.488 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:06:12.519 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:06:12.558 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:06:12.646 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:06:12.691 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:12.748 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:12.875 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:06:12.928 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:06:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:06:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:06:13.064 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:06:13.121 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:13.159 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:06:13.267 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:06:13.324 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:06:13.399 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:13.399 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:13.399 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:06:13.399 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:13.399 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:13.399 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:13.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2 2024-03-25T18:06:14.042 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:06:14.087 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:06:14.121 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:06:14.126 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:14.169 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:06:14.191 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:14.229 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:14.285 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:14.286 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:06:14.286 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:14.338 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1 2024-03-25T18:06:14.930 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:14.965 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:06:14.966 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:06:15.032 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:06:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:15.140 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados 2024-03-25T18:06:15.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:06:15.334 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:15.334 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:15.334 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:15.335 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:15.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:06:15.701 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:15.797 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:15.849 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:15.850 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:06:15.850 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:15.892 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw 2024-03-25T18:06:16.165 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:16.317 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:16.317 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:06:16.317 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:06:16.318 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:06:16.319 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:06:16.319 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:16.319 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:16.319 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:06:16.319 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-03-25T18:06:16.319 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:16.320 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-03-25T18:06:16.320 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-03-25T18:06:16.320 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:16.454 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:16.550 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:16.602 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:16.603 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:06:16.603 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:16.643 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs 2024-03-25T18:06:16.938 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T18:06:16.964 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 855 kB/s | 248 kB 00:00 2024-03-25T18:06:17.164 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 288 kB/s | 58 kB 00:00 2024-03-25T18:06:17.204 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:17.231 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T18:06:17.306 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:17.322 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-03-25T18:06:17.359 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:17.360 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:06:17.360 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:17.400 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd 2024-03-25T18:06:17.506 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 166 kB/s | 138 kB 00:00 2024-03-25T18:06:17.507 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:17.508 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-25T18:06:17.521 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:17.531 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:17.531 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:17.621 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:17.623 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:17.915 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:17.960 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:18.056 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:18.082 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:06:18.108 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:18.109 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T18:06:18.109 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:18.141 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:06:18.151 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse 2024-03-25T18:06:18.662 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:06:18.715 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:18.751 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:06:18.821 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:06:18.845 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:18.864 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:18.865 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:18.865 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:18.865 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:18.865 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:06:18.865 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:06:18.866 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:18.866 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:06:18.866 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:18.866 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:06:18.866 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:18.866 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 85 k 2024-03-25T18:06:18.866 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 231 k 2024-03-25T18:06:18.867 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:06:19.587 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 118 kB/s | 85 kB 00:00 2024-03-25T18:06:19.588 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:19.588 INFO:teuthology.orchestra.run.smithi159.stdout:Total 118 kB/s | 85 kB 00:00 2024-03-25T18:06:19.588 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:06:19.595 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:06:19.595 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:06:19.652 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:06:19.652 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:06:19.693 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:19.693 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:06:19.693 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:06:19.693 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:06:19.693 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:06:19.693 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:06:19.793 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:06:19.886 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:19.991 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:06:19.991 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:19.991 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:19.991 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:19.991 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:06:19.991 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:06:19.991 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:06:19.992 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:06:19.992 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:06:19.992 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:19.992 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:20.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-03-25T18:06:20.286 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:20.606 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:20.607 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:20.607 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:06:20.607 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:20.607 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:20.607 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:20.805 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:20.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror 2024-03-25T18:06:20.957 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:20.958 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:06:20.958 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:20.958 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:06:20.958 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:06:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:06:20.959 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:20.959 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:20.961 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:20.961 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-25T18:06:20.961 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:20.961 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 964 k 2024-03-25T18:06:20.961 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.8 M 2024-03-25T18:06:20.961 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:21.074 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 684 kB/s | 79 kB 00:00 2024-03-25T18:06:21.407 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:22 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:21.562 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:21.563 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:21.563 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:06:21.563 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:21.563 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:06:21.563 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:06:21.563 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:21.563 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:06:21.564 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:21.564 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:06:21.564 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:21.564 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 3.1 M 2024-03-25T18:06:21.564 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 13 M 2024-03-25T18:06:21.564 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:06:23.474 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 352 kB/s | 885 kB 00:02 2024-03-25T18:06:23.476 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:23.476 INFO:teuthology.orchestra.run.smithi134.stdout:Total 383 kB/s | 964 kB 00:02 2024-03-25T18:06:23.476 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:23.500 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:23.500 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:23.576 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:23.576 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:23.761 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:23.900 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:06:23.974 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:24.705 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:24.706 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:06:25.015 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:06:25.015 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:25.015 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:25.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:06:25.015 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:25.015 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:25.320 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-03-25T18:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:26.047 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:26.048 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:26.048 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T18:06:26.048 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:26.048 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:26.048 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:06:26.048 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 332 k 2024-03-25T18:06:26.049 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.5 M 2024-03-25T18:06:26.050 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:26.178 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 535 kB/s | 69 kB 00:00 2024-03-25T18:06:27.020 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 272 kB/s | 264 kB 00:00 2024-03-25T18:06:27.021 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:27.022 INFO:teuthology.orchestra.run.smithi134.stdout:Total 342 kB/s | 332 kB 00:00 2024-03-25T18:06:27.022 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:27.032 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:27.032 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:27.077 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:27.078 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:27.225 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:27.418 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:06:27.456 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:27.471 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:27.471 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:27.471 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:06:27.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:28.200 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:06:28.515 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:06:28.515 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:28.515 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:28.515 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:28.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:06:28.515 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:28.515 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:28.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-03-25T18:06:29.390 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:29.439 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 401 kB/s | 3.1 MB 00:07 2024-03-25T18:06:29.440 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:29.441 INFO:teuthology.orchestra.run.smithi159.stdout:Total 401 kB/s | 3.1 MB 00:07 2024-03-25T18:06:29.441 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:06:29.452 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:06:29.452 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:06:29.529 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:06:29.529 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:06:29.539 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:06:29.540 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:29.541 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-03-25T18:06:29.541 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-03-25T18:06:29.541 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:29.972 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:06:30.023 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:30.048 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:30.048 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:30.048 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:06:30.048 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:30.048 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:30.049 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:30.450 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 139 kB/s | 127 kB 00:00 2024-03-25T18:06:30.451 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:30.451 INFO:teuthology.orchestra.run.smithi134.stdout:Total 139 kB/s | 127 kB 00:00 2024-03-25T18:06:30.451 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:30.458 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:30.458 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:30.518 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:30.518 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:31.180 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:31.180 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:31.180 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:06:31.180 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:31.180 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:31.180 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:31.270 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:31.427 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd 2024-03-25T18:06:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:31.568 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:31.568 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:31.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-03-25T18:06:32.002 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:32.151 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:06:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:32.154 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 171 k 2024-03-25T18:06:32.154 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 494 k 2024-03-25T18:06:32.154 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:06:32.402 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:32.499 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:32.552 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:32.554 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:06:32.554 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:32.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-03-25T18:06:33.188 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:33.337 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:33.338 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:33.338 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:06:33.338 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:33.338 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:33.338 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 140 k 2024-03-25T18:06:33.339 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:33.869 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T18:06:33.870 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:33.870 INFO:teuthology.orchestra.run.smithi134.stdout:Total 59 kB/s | 31 kB 00:00 2024-03-25T18:06:33.870 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:33.874 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:33.874 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:33.905 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:33.906 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:34.042 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:34.160 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:34.301 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 80 kB/s | 171 kB 00:02 2024-03-25T18:06:34.301 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:34.302 INFO:teuthology.orchestra.run.smithi159.stdout:Total 80 kB/s | 171 kB 00:02 2024-03-25T18:06:34.302 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:06:34.311 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:06:34.311 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:06:34.372 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:06:34.372 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:06:34.585 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:34.593 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:06:34.700 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:34.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:34.838 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:34.839 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:34.839 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:34.839 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:34.839 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:34.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-03-25T18:06:35.116 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:35.395 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:06:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:35.395 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:06:35.551 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:35.648 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:35.678 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:06:35.701 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:35.702 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:06:35.702 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:35.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-03-25T18:06:36.256 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:06:36.316 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:36.410 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:06:36.410 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-03-25T18:06:36.411 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:06:36.412 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 91 M 2024-03-25T18:06:36.412 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 498 M 2024-03-25T18:06:36.412 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:06:36.413 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:36.464 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:36.465 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:06:36.466 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:36.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-03-25T18:06:37.087 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:37.183 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:37.235 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:37.236 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:06:37.236 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:37.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-03-25T18:06:37.860 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:37.966 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:38.021 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:38.023 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:06:38.023 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:38.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-03-25T18:06:38.677 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:38.776 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:38.830 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:38.831 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:06:38.832 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:38.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-03-25T18:06:39.453 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:39.550 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:39.601 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:39.602 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T18:06:39.602 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:39.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-03-25T18:06:40.218 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:40.365 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:40.366 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:40.366 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:40.366 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:40.366 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:40.366 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:06:40.366 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:40.366 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:40.367 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:40.367 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:06:40.367 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:40.367 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 85 k 2024-03-25T18:06:40.367 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 231 k 2024-03-25T18:06:40.367 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:41.331 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 88 kB/s | 85 kB 00:00 2024-03-25T18:06:41.331 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:41.332 INFO:teuthology.orchestra.run.smithi134.stdout:Total 88 kB/s | 85 kB 00:00 2024-03-25T18:06:41.332 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:41.340 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:41.340 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:41.400 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:41.400 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:41.654 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:42.095 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:42.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-03-25T18:06:43.225 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:43.380 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:43.381 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:43.381 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:06:43.381 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 13 M 2024-03-25T18:06:43.382 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:52.125 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 362 kB/s | 3.1 MB 00:08 2024-03-25T18:06:52.125 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:52.126 INFO:teuthology.orchestra.run.smithi134.stdout:Total 362 kB/s | 3.1 MB 00:08 2024-03-25T18:06:52.126 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:52.137 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:52.137 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:52.214 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:52.214 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:52.708 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:52.733 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:52.733 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:52.734 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:06:52.734 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:52.734 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:52.734 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:53.912 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:53.912 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:53.912 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:54.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-03-25T18:06:54.719 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:54.867 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:54.868 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:54.868 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:54.868 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 171 k 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-03-25T18:06:54.869 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:06:55.960 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 157 kB/s | 171 kB 00:01 2024-03-25T18:06:55.961 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:55.961 INFO:teuthology.orchestra.run.smithi134.stdout:Total 157 kB/s | 171 kB 00:01 2024-03-25T18:06:55.961 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:06:55.971 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:06:55.971 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:06:56.030 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:06:56.031 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:06:56.210 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:06:56.309 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:56.781 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:57.064 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:57.064 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:57.064 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:06:57.064 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:57.064 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:57.064 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:06:57.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:06:57.880 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:06:58.035 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:06:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-25T18:06:58.037 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:06:58.037 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 91 M 2024-03-25T18:06:58.037 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 498 M 2024-03-25T18:06:58.037 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:07:42.405 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 348 kB/s | 22 MB 01:05 2024-03-25T18:07:56.880 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 391 kB/s | 22 MB 00:58 2024-03-25T18:10:01.237 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 341 kB/s | 68 MB 03:24 2024-03-25T18:10:01.239 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:01.239 INFO:teuthology.orchestra.run.smithi159.stdout:Total 453 kB/s | 91 MB 03:24 2024-03-25T18:10:01.239 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:10:01.242 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:10:01.242 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:10:01.823 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:10:01.823 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:10:04.972 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:10:08.663 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 366 kB/s | 68 MB 03:10 2024-03-25T18:10:08.664 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:08.665 INFO:teuthology.orchestra.run.smithi134.stdout:Total 487 kB/s | 91 MB 03:10 2024-03-25T18:10:08.665 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:10:08.669 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:10:08.669 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:10:09.255 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:10:09.255 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:10:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:10:12.145 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:12.352 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:10:15.481 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:15.481 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:10:15.747 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:10:15.747 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:10:15.747 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:10:15.747 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:15.747 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:15.747 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:10:15.747 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:10:15.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:10:16.514 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:09:17 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:10:16.661 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:10:16.662 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:10:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:10:16.662 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:10:16.662 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:10:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:10:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:10:16.663 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:10:16.663 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:10:16.663 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:10:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:10:16.663 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 416 M 2024-03-25T18:10:16.663 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.7 G 2024-03-25T18:10:16.663 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:10:19.557 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:10:19.632 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:22.959 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:22.960 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:10:23.218 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:10:23.219 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:10:23.219 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:10:23.219 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:23.219 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:23.219 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:10:23.219 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:10:23.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:10:24.009 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:09:24 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:10:24.158 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:10:24.159 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:10:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:10:24.160 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 416 M 2024-03-25T18:10:24.160 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.7 G 2024-03-25T18:10:24.160 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:25:25.056 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 469 kB/s | 416 MB 15:08 2024-03-25T18:25:25.057 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:25.057 INFO:teuthology.orchestra.run.smithi159.stdout:Total 469 kB/s | 416 MB 15:08 2024-03-25T18:25:25.057 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:25:25.060 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:25:25.060 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:25:27.539 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:25:27.539 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:25:31.099 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 470 kB/s | 416 MB 15:06 2024-03-25T18:25:31.100 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:31.100 INFO:teuthology.orchestra.run.smithi134.stdout:Total 470 kB/s | 416 MB 15:06 2024-03-25T18:25:31.100 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:25:31.104 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:25:31.104 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:25:33.594 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:25:33.594 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:26:09.745 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:26:09.841 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:15.994 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:26:16.103 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:18.140 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:26:18.392 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:26:18.955 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:25:19 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:26:19.102 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:26:19.103 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:26:19.103 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:26:19.103 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:26:19.103 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:26:19.103 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:26:19.103 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:19.104 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:26:19.104 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T18:26:19.104 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:26:19.104 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:19.104 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.4 M 2024-03-25T18:26:19.104 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 6.4 M 2024-03-25T18:26:19.104 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:26:21.030 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:21.220 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 690 kB/s | 1.4 MB 00:02 2024-03-25T18:26:21.221 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:21.221 INFO:teuthology.orchestra.run.smithi159.stdout:Total 690 kB/s | 1.4 MB 00:02 2024-03-25T18:26:21.222 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:26:21.226 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:26:21.226 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:26:21.271 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:26:21.271 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:26:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:21.290 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:26:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:21.291 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:26:21.525 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:26:21.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:26:21.635 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:22.010 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:22.111 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:25:22 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:26:22.261 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:22.262 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.4 M 2024-03-25T18:26:22.263 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 6.4 M 2024-03-25T18:26:22.263 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:26:22.291 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:22.291 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:22.291 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:26:22.291 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:22.291 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:22.291 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:26:22.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:26:23.032 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:25:24 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:26:23.181 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:26:23.183 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:26:23.183 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:26:23.183 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:26:23.183 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:26:23.184 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:26:23.184 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:23.184 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:26:23.184 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:26:23.184 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:26:23.184 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:26:23.184 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 176 M 2024-03-25T18:26:23.185 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 753 M 2024-03-25T18:26:23.185 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:26:26.372 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 356 kB/s | 1.4 MB 00:04 2024-03-25T18:26:26.373 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:26.373 INFO:teuthology.orchestra.run.smithi134.stdout:Total 355 kB/s | 1.4 MB 00:04 2024-03-25T18:26:26.373 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:26:26.377 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:26:26.377 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:26:26.423 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:26:26.423 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:26:26.689 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:26:26.796 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:27.228 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:27.501 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:27.502 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:27.502 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:26:27.502 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:27.502 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:27.502 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:26:27.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:26:28.227 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:25:28 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:26:28.376 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:26:28.377 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:26:28.377 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:26:28.377 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:26:28.377 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 176 M 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 753 M 2024-03-25T18:26:28.378 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:30:06.512 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 825 kB/s | 176 MB 03:38 2024-03-25T18:30:06.513 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:06.513 INFO:teuthology.orchestra.run.smithi134.stdout:Total 825 kB/s | 176 MB 03:38 2024-03-25T18:30:06.513 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:30:06.516 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:30:06.516 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:30:07.574 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:30:07.575 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T18:30:25.559 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T18:30:25.875 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:30:30.563 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:30:30.819 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:30:30.819 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:30:30.819 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T18:30:30.819 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.819 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:30:30.819 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T18:30:31.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:29:31 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T18:30:31.740 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T18:30:31.741 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T18:30:31.742 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 672 M 2024-03-25T18:30:31.742 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.1 G 2024-03-25T18:30:31.742 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T18:31:30.234 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 586 kB/s | 176 MB 05:07 2024-03-25T18:31:30.235 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:30.235 INFO:teuthology.orchestra.run.smithi159.stdout:Total 586 kB/s | 176 MB 05:07 2024-03-25T18:31:30.235 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T18:31:30.238 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T18:31:30.238 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T18:31:31.296 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T18:31:31.296 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T18:31:49.556 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T18:31:49.748 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:31:54.428 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:31:54.696 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:31:54.696 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:31:54.696 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T18:31:54.696 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:54.696 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:31:54.696 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T18:31:54.888 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:31:55.450 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:30:56 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T18:31:55.598 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T18:31:55.598 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 672 M 2024-03-25T18:31:55.599 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 3.1 G 2024-03-25T18:31:55.600 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T18:59:14.028 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 400 kB/s | 672 MB 28:42 2024-03-25T18:59:14.029 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:14.029 INFO:teuthology.orchestra.run.smithi134.stdout:Total 400 kB/s | 672 MB 28:42 2024-03-25T18:59:14.029 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T18:59:14.033 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T18:59:14.034 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T18:59:18.005 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T18:59:18.005 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:00:29.149 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:00:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:00:39.040 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:00:39.298 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:00:39.298 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:00:39.299 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:00:39.299 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:39.299 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:00:39.299 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:00:39.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:00:40.144 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:59:40 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:00:40.241 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:00:40.293 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:00:40.294 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T19:00:40.294 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:00:40.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:00:40.928 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:59:40 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:00:41.078 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 20 M 2024-03-25T19:00:41.079 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 103 M 2024-03-25T19:00:41.080 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:01:34.683 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 376 kB/s | 20 MB 00:53 2024-03-25T19:01:34.684 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:01:34.684 INFO:teuthology.orchestra.run.smithi134.stdout:Total 376 kB/s | 20 MB 00:53 2024-03-25T19:01:34.684 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:01:34.688 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:01:34.688 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:01:34.839 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:01:34.839 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:01:37.065 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:01:37.238 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:01:38.247 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:01:38.514 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:01:38.515 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:01:38.515 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:01:38.515 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:01:38.515 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:01:38.515 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:01:38.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:01:39.266 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:00:39 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:01:39.414 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:01:39.415 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-03-25T19:01:39.416 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 173 M 2024-03-25T19:01:39.416 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:03:20.261 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 365 kB/s | 672 MB 31:24 2024-03-25T19:03:20.261 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:20.261 INFO:teuthology.orchestra.run.smithi159.stdout:Total 365 kB/s | 672 MB 31:24 2024-03-25T19:03:20.261 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:03:20.265 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:03:20.265 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:03:24.211 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:03:24.212 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:03:28.628 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 447 kB/s | 48 MB 01:49 2024-03-25T19:03:28.629 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:28.629 INFO:teuthology.orchestra.run.smithi134.stdout:Total 447 kB/s | 48 MB 01:49 2024-03-25T19:03:28.629 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:03:28.633 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:03:28.633 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:03:28.946 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:03:28.947 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:03:33.688 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:03:33.785 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:35.109 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:35.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:03:35.378 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:03:35.379 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:03:35.379 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:03:35.379 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:03:35.379 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:03:35.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:03:36.140 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:02:36 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:03:36.288 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:03:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-03-25T19:03:36.290 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 449 M 2024-03-25T19:03:36.290 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:04:35.023 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:04:35.646 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:44.626 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:44.896 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:44.897 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:04:44.897 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:04:44.897 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:44.897 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:04:44.897 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:04:45.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:04:45.742 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:03:46 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:04:45.839 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:04:45.890 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:04:45.891 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T19:04:45.892 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:04:45.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:04:46.520 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:03:47 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:04:46.668 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:04:46.668 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:04:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:04:46.670 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 20 M 2024-03-25T19:04:46.670 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 103 M 2024-03-25T19:04:46.670 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:05:33.959 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 427 kB/s | 20 MB 00:47 2024-03-25T19:05:33.959 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:33.960 INFO:teuthology.orchestra.run.smithi159.stdout:Total 427 kB/s | 20 MB 00:47 2024-03-25T19:05:33.960 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:05:33.964 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:05:33.964 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:05:34.113 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:05:34.113 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:05:36.342 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:05:36.449 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:37.419 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:05:37.676 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:05:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:37.676 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:05:37.676 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:05:37.857 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:05:38.422 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:04:39 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:05:38.572 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:05:38.573 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 48 M 2024-03-25T19:05:38.574 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 173 M 2024-03-25T19:05:38.574 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:07:07.608 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 549 kB/s | 48 MB 01:29 2024-03-25T19:07:07.609 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:07.609 INFO:teuthology.orchestra.run.smithi159.stdout:Total 549 kB/s | 48 MB 01:29 2024-03-25T19:07:07.609 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:07:07.613 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:07:07.613 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:07:07.925 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:07:07.925 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:07:12.651 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:07:12.781 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:14.134 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:14.403 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:14.403 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:07:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:07:14.403 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:14.403 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:07:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:07:14.573 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:07:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:06:16 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:07:15.289 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:07:15.289 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:07:15.289 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:07:15.289 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:07:15.289 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:07:15.289 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:07:15.290 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:07:15.290 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:07:15.290 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:07:15.290 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:07:15.290 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:07:15.290 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 99 M 2024-03-25T19:07:15.290 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 449 M 2024-03-25T19:07:15.291 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:07:50.020 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 398 kB/s | 99 MB 04:13 2024-03-25T19:07:50.021 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:50.021 INFO:teuthology.orchestra.run.smithi134.stdout:Total 398 kB/s | 99 MB 04:13 2024-03-25T19:07:50.021 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:07:50.024 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:07:50.024 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:07:50.635 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:07:50.635 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:08:00.924 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:08:01.016 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:08:03.861 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:08:04.123 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:08:04.123 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:08:04.123 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:08:04.123 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:08:04.123 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:08:04.123 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:08:04.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:08:04.865 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:07:04 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:08:05.014 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:08:05.015 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:08:05.016 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 287 M 2024-03-25T19:08:05.016 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.2 G 2024-03-25T19:08:05.016 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:10:21.387 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 542 kB/s | 99 MB 03:06 2024-03-25T19:10:21.387 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:21.387 INFO:teuthology.orchestra.run.smithi159.stdout:Total 542 kB/s | 99 MB 03:06 2024-03-25T19:10:21.388 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:10:21.390 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:10:21.391 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:10:21.998 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:10:21.998 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:10:32.308 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:10:32.400 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:35.305 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:35.581 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:35.581 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:10:35.581 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:10:35.581 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:35.582 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:10:35.582 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:10:35.759 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:10:36.320 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:09:37 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:10:36.469 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:10:36.469 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:10:36.469 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:10:36.469 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:10:36.469 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 287 M 2024-03-25T19:10:36.470 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.2 G 2024-03-25T19:10:36.471 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:21:23.434 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 368 kB/s | 287 MB 13:18 2024-03-25T19:21:23.435 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:23.435 INFO:teuthology.orchestra.run.smithi134.stdout:Total 368 kB/s | 287 MB 13:18 2024-03-25T19:21:23.435 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:21:23.438 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:21:23.438 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:21:25.141 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:21:25.141 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:21:45.625 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 439 kB/s | 287 MB 11:09 2024-03-25T19:21:45.626 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:45.626 INFO:teuthology.orchestra.run.smithi159.stdout:Total 439 kB/s | 287 MB 11:09 2024-03-25T19:21:45.626 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:21:45.629 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:21:45.629 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:21:47.338 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:21:47.338 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:21:54.767 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:21:54.870 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:02.234 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:02.516 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:02.516 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:02.516 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:22:02.517 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:02.517 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:02.517 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:22:02.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:22:03.300 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:21:03 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:22:03.451 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:22:03.451 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:03.451 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:22:03.451 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:03.451 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:22:03.451 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:22:03.451 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:03.452 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:22:03.452 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:03.452 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:22:03.452 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:03.452 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.9 M 2024-03-25T19:22:03.452 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 28 M 2024-03-25T19:22:03.452 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:22:12.023 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 710 kB/s | 5.9 MB 00:08 2024-03-25T19:22:12.024 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:12.025 INFO:teuthology.orchestra.run.smithi134.stdout:Total 710 kB/s | 5.9 MB 00:08 2024-03-25T19:22:12.025 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:22:12.029 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:22:12.029 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:22:12.101 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:22:12.101 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:22:12.840 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:22:12.955 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:13.795 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:13.795 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:13.795 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:22:13.795 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:13.795 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:13.795 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:22:14.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:22:14.587 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:21:14 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:22:14.740 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 109 k 2024-03-25T19:22:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 440 k 2024-03-25T19:22:14.742 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:22:15.551 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 134 kB/s | 109 kB 00:00 2024-03-25T19:22:15.552 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:15.553 INFO:teuthology.orchestra.run.smithi134.stdout:Total 134 kB/s | 109 kB 00:00 2024-03-25T19:22:15.553 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:22:15.556 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:22:15.556 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:22:15.592 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:22:15.592 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:22:15.707 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:22:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:16.195 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:16.471 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:16.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:16.471 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:22:16.471 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:16.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:16.471 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:22:16.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:22:16.754 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:22:16.845 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:17.252 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:21:17 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:22:17.402 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:22:17.402 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:17.402 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:22:17.402 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:17.402 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:22:17.402 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.4 M 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 25 M 2024-03-25T19:22:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:22:23.154 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 953 kB/s | 5.4 MB 00:05 2024-03-25T19:22:23.155 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:23.155 INFO:teuthology.orchestra.run.smithi134.stdout:Total 953 kB/s | 5.4 MB 00:05 2024-03-25T19:22:23.155 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:22:23.159 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:22:23.160 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:22:23.229 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:22:23.230 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:22:23.898 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:22:24.011 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:22:24.358 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:24.578 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:22:24.641 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:24.641 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:24.642 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:22:24.642 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:24.642 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:24.642 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:22:24.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:22:24.844 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:24.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:22:24.844 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:24.844 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:24.844 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:22:24.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:22:25.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2-debuginfo 2024-03-25T19:22:25.411 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:21:26 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:22:25.560 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:25.561 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 5.9 M 2024-03-25T19:22:25.562 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 28 M 2024-03-25T19:22:25.562 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:22:25.622 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:21:25 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:22:25.770 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:22:25.771 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:22:25.772 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:22:25.772 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 44 M 2024-03-25T19:22:25.772 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 253 M 2024-03-25T19:22:25.772 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:22:34.348 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 693 kB/s | 5.9 MB 00:08 2024-03-25T19:22:34.349 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:34.349 INFO:teuthology.orchestra.run.smithi159.stdout:Total 692 kB/s | 5.9 MB 00:08 2024-03-25T19:22:34.349 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:22:34.353 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:22:34.354 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:22:34.425 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:22:34.426 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:22:35.162 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:22:35.264 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:35.887 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:36.164 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:36.164 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:36.164 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:22:36.164 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:36.164 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:36.164 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:22:36.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:22:36.929 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:21:37 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:22:37.082 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:22:37.082 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:22:37.083 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:37.084 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 109 k 2024-03-25T19:22:37.084 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 440 k 2024-03-25T19:22:37.084 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:22:37.901 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 133 kB/s | 109 kB 00:00 2024-03-25T19:22:37.901 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:37.902 INFO:teuthology.orchestra.run.smithi159.stdout:Total 133 kB/s | 109 kB 00:00 2024-03-25T19:22:37.902 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:22:37.905 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:22:37.905 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:22:37.941 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:22:37.942 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:22:38.055 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:22:38.158 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:38.597 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:38.871 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:22:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:38.871 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:38.871 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:22:39.052 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:22:39.616 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:21:40 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:22:39.765 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:22:39.765 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:22:39.766 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:39.767 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 5.4 M 2024-03-25T19:22:39.767 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 25 M 2024-03-25T19:22:39.767 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:22:55.719 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 344 kB/s | 5.4 MB 00:15 2024-03-25T19:22:55.719 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:55.720 INFO:teuthology.orchestra.run.smithi159.stdout:Total 344 kB/s | 5.4 MB 00:15 2024-03-25T19:22:55.720 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:22:55.724 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:22:55.724 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:22:55.793 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:22:55.793 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:22:56.458 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:22:56.551 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:22:57.088 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:22:57.359 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:22:57.359 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:57.359 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:22:57.360 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:57.360 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:57.360 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:22:57.545 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2-debuginfo 2024-03-25T19:22:58.110 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:21:59 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:22:58.258 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:22:58.259 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:22:58.259 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:58.259 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:22:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:22:58.259 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:58.259 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:22:58.260 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:22:58.260 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:22:58.260 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:22:58.260 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 44 M 2024-03-25T19:22:58.260 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 253 M 2024-03-25T19:22:58.260 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:23:52.513 INFO:teuthology.orchestra.run.smithi134.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 523 kB/s | 44 MB 01:26 2024-03-25T19:23:52.514 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:52.514 INFO:teuthology.orchestra.run.smithi134.stdout:Total 523 kB/s | 44 MB 01:26 2024-03-25T19:23:52.515 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:23:52.518 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:23:52.518 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:23:52.817 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:23:52.817 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:23:57.907 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:23:57.995 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:59.732 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:59.995 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:59.996 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:23:59.996 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:23:59.996 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:59.996 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:23:59.996 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:24:00.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:24:00.773 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:23:00 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:24:00.925 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:24:00.926 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:24:00.926 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:24:00.926 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:24:00.926 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:24:00.926 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 52 M 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 308 M 2024-03-25T19:24:00.927 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:24:02.680 INFO:teuthology.orchestra.run.smithi159.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 704 kB/s | 44 MB 01:04 2024-03-25T19:24:02.681 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:02.681 INFO:teuthology.orchestra.run.smithi159.stdout:Total 704 kB/s | 44 MB 01:04 2024-03-25T19:24:02.682 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:24:02.685 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:24:02.685 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:24:02.984 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:24:03.136 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:24:08.035 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:24:08.121 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:10.005 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:10.272 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:10.272 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:24:10.272 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:24:10.272 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:10.272 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:24:10.272 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:24:10.455 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:24:11.017 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:23:12 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:24:11.165 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:24:11.166 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 52 M 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 308 M 2024-03-25T19:24:11.167 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:25:05.465 INFO:teuthology.orchestra.run.smithi134.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 823 kB/s | 52 MB 01:04 2024-03-25T19:25:05.466 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:05.466 INFO:teuthology.orchestra.run.smithi134.stdout:Total 823 kB/s | 52 MB 01:04 2024-03-25T19:25:05.467 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:25:05.471 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:25:05.471 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:25:05.817 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:25:05.817 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:25:11.803 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:25:11.905 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:12.525 INFO:teuthology.orchestra.run.smithi159.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 865 kB/s | 52 MB 01:01 2024-03-25T19:25:12.526 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:12.526 INFO:teuthology.orchestra.run.smithi159.stdout:Total 865 kB/s | 52 MB 01:01 2024-03-25T19:25:12.527 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:25:12.530 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:25:12.530 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:25:12.874 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:25:12.874 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:25:14.036 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:14.298 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:14.298 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:14.298 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:25:14.298 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:14.298 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:14.298 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:25:14.504 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:25:15.064 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:24:15 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:25:15.215 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:25:15.216 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:15.216 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:25:15.216 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 615 k 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.1 M 2024-03-25T19:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:25:17.593 INFO:teuthology.orchestra.run.smithi134.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 259 kB/s | 615 kB 00:02 2024-03-25T19:25:17.594 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:17.594 INFO:teuthology.orchestra.run.smithi134.stdout:Total 259 kB/s | 615 kB 00:02 2024-03-25T19:25:17.595 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:25:17.598 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:25:17.599 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:25:17.640 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:25:17.640 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:25:17.805 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:25:17.916 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:18.337 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:18.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:18.606 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:18.606 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:25:18.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:18.606 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:18.607 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:25:18.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:25:18.868 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:25:19.048 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:19.384 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:24:19 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 771 k 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.7 M 2024-03-25T19:25:19.537 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:25:21.273 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:21.535 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:21.535 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:21.535 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:25:21.535 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:21.535 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:21.535 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:25:21.712 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:25:21.905 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 326 kB/s | 771 kB 00:02 2024-03-25T19:25:21.905 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:21.906 INFO:teuthology.orchestra.run.smithi134.stdout:Total 325 kB/s | 771 kB 00:02 2024-03-25T19:25:21.906 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:25:21.910 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:25:21.910 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:25:21.953 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:25:21.953 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:25:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:25:22.276 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:24:23 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:25:22.287 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:22.426 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:25:22.427 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:22.428 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 615 k 2024-03-25T19:25:22.428 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.1 M 2024-03-25T19:25:22.428 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:25:22.672 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:22.941 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:25:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:22.941 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:22.941 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:25:23.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:25:23.729 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:24:23 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:25:23.877 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:25:23.878 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:23.879 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:25:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:23.879 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.1 M 2024-03-25T19:25:23.879 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 4.3 M 2024-03-25T19:25:23.879 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:25:25.403 INFO:teuthology.orchestra.run.smithi159.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 207 kB/s | 615 kB 00:02 2024-03-25T19:25:25.403 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:25.403 INFO:teuthology.orchestra.run.smithi159.stdout:Total 207 kB/s | 615 kB 00:02 2024-03-25T19:25:25.404 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:25:25.408 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:25:25.408 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:25:25.448 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:25:25.449 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:25:25.613 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:25:25.734 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:26.139 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:26.401 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:25:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:26.401 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:26.401 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:25:26.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:25:26.649 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 415 kB/s | 1.1 MB 00:02 2024-03-25T19:25:26.650 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:26.651 INFO:teuthology.orchestra.run.smithi134.stdout:Total 415 kB/s | 1.1 MB 00:02 2024-03-25T19:25:26.651 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:25:26.655 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:25:26.655 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:25:26.698 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:25:26.699 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:25:26.914 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:25:27.023 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:27.153 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:24:28 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:25:27.306 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 771 k 2024-03-25T19:25:27.307 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.7 M 2024-03-25T19:25:27.308 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:25:27.439 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:27.697 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:27.698 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:27.698 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:25:27.698 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:27.698 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:27.698 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:25:27.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:25:28.463 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:24:28 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:25:28.612 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:25:28.612 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:28.612 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:25:28.612 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:28.613 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 337 k 2024-03-25T19:25:28.614 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.0 M 2024-03-25T19:25:28.614 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:25:29.581 INFO:teuthology.orchestra.run.smithi159.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 339 kB/s | 771 kB 00:02 2024-03-25T19:25:29.581 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:29.582 INFO:teuthology.orchestra.run.smithi159.stdout:Total 339 kB/s | 771 kB 00:02 2024-03-25T19:25:29.582 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:25:29.586 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:25:29.586 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:25:29.619 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 336 kB/s | 337 kB 00:01 2024-03-25T19:25:29.619 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:29.620 INFO:teuthology.orchestra.run.smithi134.stdout:Total 335 kB/s | 337 kB 00:01 2024-03-25T19:25:29.620 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:25:29.623 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:25:29.623 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:25:29.628 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:25:29.628 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:25:29.659 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:25:29.659 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:25:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:25:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:25:29.904 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:29.913 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:30.277 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:30.290 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:30.545 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:30.545 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:30.545 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:25:30.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:30.546 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:30.546 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:25:30.552 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:30.552 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:30.552 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:25:30.552 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:30.552 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:30.552 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:25:30.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:25:30.791 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:25:31.322 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:24:31 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:25:31.360 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:24:32 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:25:31.472 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:25:31.472 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:31.472 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:25:31.472 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 509 k 2024-03-25T19:25:31.473 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.5 M 2024-03-25T19:25:31.474 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:25:31.513 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:25:31.513 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:25:31.513 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:31.513 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:25:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.1 M 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 4.3 M 2024-03-25T19:25:31.514 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:25:33.697 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 229 kB/s | 509 kB 00:02 2024-03-25T19:25:33.698 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:33.699 INFO:teuthology.orchestra.run.smithi134.stdout:Total 229 kB/s | 509 kB 00:02 2024-03-25T19:25:33.699 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:25:33.703 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:25:33.703 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:25:33.744 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:25:33.744 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:25:33.908 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:25:34.004 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:34.108 INFO:teuthology.orchestra.run.smithi159.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 444 kB/s | 1.1 MB 00:02 2024-03-25T19:25:34.109 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:34.110 INFO:teuthology.orchestra.run.smithi159.stdout:Total 443 kB/s | 1.1 MB 00:02 2024-03-25T19:25:34.110 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:25:34.114 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:25:34.114 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:25:34.158 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:25:34.159 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:25:34.391 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:25:34.417 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:34.687 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:34.687 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:34.688 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:25:34.688 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:34.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:34.688 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:25:34.891 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:34.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:25:35.142 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:35.142 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:35.142 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:25:35.142 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:35.142 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:35.142 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:25:35.348 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:25:35.510 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:24:35 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:25:35.658 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:25:35.659 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:35.659 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:25:35.659 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:35.659 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 52 M 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 298 M 2024-03-25T19:25:35.660 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:25:35.916 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:24:36 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:25:36.066 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:25:36.066 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:36.066 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:25:36.066 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:36.066 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 337 k 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.0 M 2024-03-25T19:25:36.067 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:25:37.579 INFO:teuthology.orchestra.run.smithi159.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 223 kB/s | 337 kB 00:01 2024-03-25T19:25:37.579 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:37.579 INFO:teuthology.orchestra.run.smithi159.stdout:Total 223 kB/s | 337 kB 00:01 2024-03-25T19:25:37.580 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:25:37.584 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:25:37.584 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:25:37.623 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:25:37.624 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:25:37.761 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:25:37.860 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:38.244 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:38.507 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:38.507 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:38.507 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:25:38.507 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:38.507 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:38.507 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:25:38.701 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:25:39.265 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:24:40 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:25:39.413 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:25:39.414 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:39.415 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 509 k 2024-03-25T19:25:39.415 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.5 M 2024-03-25T19:25:39.415 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:25:41.398 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 257 kB/s | 509 kB 00:01 2024-03-25T19:25:41.398 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:41.399 INFO:teuthology.orchestra.run.smithi159.stdout:Total 257 kB/s | 509 kB 00:01 2024-03-25T19:25:41.399 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:25:41.403 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:25:41.403 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:25:41.444 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:25:41.445 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:25:41.610 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:25:41.721 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:42.102 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:42.369 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:42.369 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:42.369 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:25:42.369 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:42.369 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:42.369 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:25:42.544 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:25:43.109 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:24:44 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:25:43.259 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:25:43.260 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:25:43.260 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:25:43.260 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 52 M 2024-03-25T19:25:43.260 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 298 M 2024-03-25T19:25:43.260 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:26:46.517 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 745 kB/s | 52 MB 01:10 2024-03-25T19:26:46.517 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:46.518 INFO:teuthology.orchestra.run.smithi134.stdout:Total 745 kB/s | 52 MB 01:10 2024-03-25T19:26:46.518 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:26:46.521 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:26:46.521 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:26:46.856 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:26:46.856 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:26:52.766 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:26:53.015 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:26:55.051 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:26:55.317 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:26:55.317 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:26:55.317 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:26:55.317 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:55.317 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:26:55.318 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:26:55.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:26:56.088 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:25:56 ago on Mon 25 Mar 2024 06:01:00 PM UTC. 2024-03-25T19:26:56.239 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T19:26:56.239 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T19:26:56.240 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:26:56.241 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.3 M 2024-03-25T19:26:56.241 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 5.5 M 2024-03-25T19:26:56.241 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T19:27:00.276 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 318 kB/s | 1.3 MB 00:04 2024-03-25T19:27:00.277 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:00.277 INFO:teuthology.orchestra.run.smithi134.stdout:Total 318 kB/s | 1.3 MB 00:04 2024-03-25T19:27:00.277 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T19:27:00.281 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T19:27:00.282 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T19:27:00.327 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T19:27:00.327 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T19:27:00.588 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T19:27:00.693 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:01.098 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:01.400 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T19:27:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:01.400 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T19:27:01.613 DEBUG:teuthology.parallel:result is None 2024-03-25T19:27:02.769 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 664 kB/s | 52 MB 01:19 2024-03-25T19:27:02.770 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:02.771 INFO:teuthology.orchestra.run.smithi159.stdout:Total 664 kB/s | 52 MB 01:19 2024-03-25T19:27:02.771 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:27:02.774 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:27:02.774 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:27:03.110 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:27:03.111 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:27:09.028 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:27:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:27:11.179 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:27:11.454 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:27:11.454 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:11.454 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:27:11.454 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:11.454 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:11.454 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:27:11.631 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:27:12.198 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:26:13 ago on Mon 25 Mar 2024 06:00:59 PM UTC. 2024-03-25T19:27:12.348 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T19:27:12.348 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:27:12.348 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T19:27:12.349 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:12.350 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.3 M 2024-03-25T19:27:12.350 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 5.5 M 2024-03-25T19:27:12.350 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T19:27:17.875 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 232 kB/s | 1.3 MB 00:05 2024-03-25T19:27:17.876 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:17.876 INFO:teuthology.orchestra.run.smithi159.stdout:Total 232 kB/s | 1.3 MB 00:05 2024-03-25T19:27:17.877 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T19:27:17.881 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T19:27:17.881 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T19:27:17.926 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T19:27:17.927 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T19:27:18.169 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T19:27:18.265 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:18.682 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:18.942 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:18.943 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:18.943 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T19:27:18.943 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:18.943 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:18.943 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T19:27:19.132 DEBUG:teuthology.parallel:result is None 2024-03-25T19:27:19.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:27:19.366 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:27:19.390 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:27:19.391 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:27:19.391 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:27:19.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:27:19.541 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:27:19.561 INFO:teuthology.orchestra.run.smithi159.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:27:19.562 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:27:19.562 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:27:19.563 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:27:19.564 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:19.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:27:19.595 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:19.595 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:27:19.629 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:27:19.629 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:19.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:27:19.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:27:19.737 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:19.737 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:27:19.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:27:19.832 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:27:19.832 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:19.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:27:19.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:27:19.932 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:19.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:27:19.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:27:20.026 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:27:20.026 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:20.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:27:20.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:27:20.124 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:20.124 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:27:20.150 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:27:20.216 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:27:20.299 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:27:20.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-03-25T19:27:20.301 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-03-25T19:27:20.328 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:27:20.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:27:20.371 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:27:20.398 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:27:20.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:27:20.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:27:20.476 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:27:20.476 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:27:20.477 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621314', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA2kYWyeLWstO2L6gPtqoaupxwqZeJGQBC0Y1ga9le3tLy/Ar4QYnSk9MoRJYwdqZ0bE0cpZ8jOK54Ch+/4cOEA=', 'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGX+Tafmgyf1WOPecYLO/4QNjiuEFjSTSXj4gKaNTmmczFBACEc92+GuTsy4TP3mDPyVmz5aT9XP/oSlYbNjtE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:27:20.477 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:27:20.507 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:27:20.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:27:20.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:27:20.589 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:20.589 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:27:20.642 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:27:20.642 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-03-25T19:27:20.696 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:27:20.696 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:20.696 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:27:20.697 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:20.697 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:20.697 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 18:06:52.971218944 +0000 2024-03-25T19:27:20.697 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 17:59:52.305328105 +0000 2024-03-25T19:27:20.697 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 17:59:52.305328105 +0000 2024-03-25T19:27:20.697 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 17:59:52.305328105 +0000 2024-03-25T19:27:20.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:27:20.762 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T19:27:20.762 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T19:27:20.762 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000238291 s, 2.1 MB/s 2024-03-25T19:27:20.763 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:27:20.819 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-03-25T19:27:20.874 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:27:20.874 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:20.875 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:27:20.875 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:20.875 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:20.875 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 18:06:52.971218944 +0000 2024-03-25T19:27:20.875 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 17:59:52.636320371 +0000 2024-03-25T19:27:20.875 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 17:59:52.636320371 +0000 2024-03-25T19:27:20.875 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 17:59:52.636320371 +0000 2024-03-25T19:27:20.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:27:20.939 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T19:27:20.940 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T19:27:20.940 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000212338 s, 2.4 MB/s 2024-03-25T19:27:20.941 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:27:20.997 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 18:06:52.971218944 +0000 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 17:59:52.977312404 +0000 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 17:59:52.977312404 +0000 2024-03-25T19:27:21.051 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 17:59:52.977312404 +0000 2024-03-25T19:27:21.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:27:21.116 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T19:27:21.116 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T19:27:21.116 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000242195 s, 2.1 MB/s 2024-03-25T19:27:21.117 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:27:21.173 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 18:06:52.972218921 +0000 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 17:59:53.314304530 +0000 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 17:59:53.314304530 +0000 2024-03-25T19:27:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 17:59:53.314304530 +0000 2024-03-25T19:27:21.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:27:21.293 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T19:27:21.293 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T19:27:21.293 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000247176 s, 2.1 MB/s 2024-03-25T19:27:21.295 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:27:21.350 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:21.350 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:21.350 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:27:21.366 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:27:21.366 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 18:06:30.288748645 +0000 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 17:59:48.196961578 +0000 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 17:59:48.196961578 +0000 2024-03-25T19:27:21.422 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 17:59:48.196961578 +0000 2024-03-25T19:27:21.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:27:21.488 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T19:27:21.488 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T19:27:21.488 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000212629 s, 2.4 MB/s 2024-03-25T19:27:21.490 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:27:21.546 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-03-25T19:27:21.601 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:27:21.601 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:21.601 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T19:27:21.601 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:21.601 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:21.602 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 18:06:30.288748645 +0000 2024-03-25T19:27:21.602 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 17:59:48.535954807 +0000 2024-03-25T19:27:21.602 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 17:59:48.535954807 +0000 2024-03-25T19:27:21.602 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 17:59:48.535954807 +0000 2024-03-25T19:27:21.602 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:27:21.668 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T19:27:21.668 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T19:27:21.668 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000196968 s, 2.6 MB/s 2024-03-25T19:27:21.669 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:27:21.725 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-03-25T19:27:21.780 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:27:21.780 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:21.780 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:27:21.781 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:21.781 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:21.781 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 18:06:30.288748645 +0000 2024-03-25T19:27:21.781 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 17:59:48.887947776 +0000 2024-03-25T19:27:21.781 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 17:59:48.887947776 +0000 2024-03-25T19:27:21.781 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 17:59:48.887947776 +0000 2024-03-25T19:27:21.781 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:27:21.846 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T19:27:21.846 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T19:27:21.846 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000214858 s, 2.4 MB/s 2024-03-25T19:27:21.847 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:27:21.903 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-03-25T19:27:21.957 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:27:21.957 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:27:21.957 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:27:21.957 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:27:21.958 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:27:21.958 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 18:06:30.289748626 +0000 2024-03-25T19:27:21.958 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 17:59:49.225941024 +0000 2024-03-25T19:27:21.958 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 17:59:49.225941024 +0000 2024-03-25T19:27:21.958 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 17:59:49.225941024 +0000 2024-03-25T19:27:21.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:27:22.023 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T19:27:22.023 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T19:27:22.023 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000234899 s, 2.2 MB/s 2024-03-25T19:27:22.024 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:27:22.080 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:22.080 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:27:22.080 INFO:tasks.ceph:Generating config... 2024-03-25T19:27:22.081 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:27:22.081 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:27:22.081 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:27:22.081 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:27:22.081 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:27:22.081 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:27:22.082 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:27:22.082 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:27:22.082 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:27:22.082 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:27:22.082 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:27:22.083 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:27:22.083 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:27:22.084 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:27:22.085 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:27:22.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:27:22.148 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:27:22.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:27:22.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:27:22.221 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.134:6789]'), ('mon.c', '[v1:172.21.15.134:6790]'), ('mon.b', '[v1:172.21.15.159:6789]')] 2024-03-25T19:27:22.222 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.134:6789],[v1:172.21.15.134:6790],[v1:172.21.15.159:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:27:22.222 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:22.223 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:27:22.276 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.134:6789]' --addv c '[v1:172.21.15.134:6790]' --addv b '[v1:172.21.15.159:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: generated fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = quincy 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 0 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:min_mon_release 17 (quincy) 2024-03-25T19:27:22.362 INFO:teuthology.orchestra.run.smithi134.stdout:election_strategy: 3 2024-03-25T19:27:22.363 INFO:teuthology.orchestra.run.smithi134.stdout:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:27:22.363 INFO:teuthology.orchestra.run.smithi134.stdout:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:27:22.363 INFO:teuthology.orchestra.run.smithi134.stdout:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:27:22.363 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:27:22.364 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:27:22.377 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 66efaa2b-ef2e-4206-b9a2-97b6c1780495... 2024-03-25T19:27:22.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:27:22.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-03-25T19:27:22.468 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-03-25T19:27:22.469 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-03-25T19:27:22.470 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = "[v1:172.21.15.134:6789],[v1:172.21.15.134:6790],[v1:172.21.15.159:6789]" 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: mon election default strategy = 3 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: ms bind msgr2 = False 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: ms type = async 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: osd heartbeat grace = 80 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-03-25T19:27:22.471 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-03-25T19:27:22.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: bdev async discard = True 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: bdev enable discard = True 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore allocator = stupid 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-03-25T19:27:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore zero block detection = True 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: debug monc = 20 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 10 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: osd class default list = * 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: osd class load list = * 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: osd client watch timeout = 120 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify cached snaps = True 2024-03-25T19:27:22.474 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify missing on start = True 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd fast shutdown = False 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd max markdown count = 1000 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock override recovery settings = True 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd op queue = debug_random 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd op queue cut off = debug_random 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: osd shutdown pgref assert = True 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:27:22.475 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:27:22.476 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon min osdmap epochs = 50 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd crush smoke test = False 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune interval = 2 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune min = 15 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon scrub interval = 300 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-03-25T19:27:22.477 INFO:teuthology.orchestra.run.smithi134.stdout: paxos service trim min = 10 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: debug monc = 20 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: debug objecter = 20 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout: debug rados = 20 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.c] 2024-03-25T19:27:22.478 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.b] 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-03-25T19:27:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:27:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-03-25T19:27:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "[v1:172.21.15.134:6789],[v1:172.21.15.134:6790],[v1:172.21.15.159:6789]" 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: mon client directed command retry = 5 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: mon election default strategy = 3 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: ms bind msgr2 = False 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject socket failures = 5000 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: ms type = async 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: osd heartbeat grace = 80 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:27:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug trim objects = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: bdev async discard = True 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: bdev enable discard = True 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore allocator = stupid 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore zero block detection = True 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 1 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 1 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: debug monc = 20 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2024-03-25T19:27:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd class default list = * 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd class load list = * 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd client watch timeout = 120 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify cached snaps = True 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify missing on start = True 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd fast shutdown = False 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd max markdown count = 1000 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock override recovery settings = True 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd op queue = debug_random 2024-03-25T19:27:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: osd op queue cut off = debug_random 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: osd shutdown pgref assert = True 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-25T19:27:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:27:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon min osdmap epochs = 50 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd crush smoke test = False 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune interval = 2 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune min = 15 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon scrub interval = 300 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on pool no app = False 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: paxos service trim min = 10 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-03-25T19:27:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: debug monc = 20 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: debug objecter = 20 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: debug rados = 20 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2024-03-25T19:27:22.489 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2024-03-25T19:27:22.490 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:27:22.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:27:22.545 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:27:22.545 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:22.545 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:27:22.560 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:22.560 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:27:22.615 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:27:22.615 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:22.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:27:22.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:27:22.695 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:22.695 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:22.749 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:27:22.749 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:22.749 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:27:22.749 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:27:22.790 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:22.790 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:22.845 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:27:22.845 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:27:22.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:27:22.899 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:27:22.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:27:22.961 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:27:22.963 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:27:22.963 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:27:22.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:27:23.005 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:27:23.021 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:27:23.064 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:27:23.079 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:27:23.079 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:27:23.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:27:23.106 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:23.106 INFO:tasks.ceph:role: osd.0 2024-03-25T19:27:23.106 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:27:23.106 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:27:23.180 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:23.180 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:23.180 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:23.181 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:23.181 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:23.181 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:23.181 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:23.687 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T19:27:23.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T19:27:23.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:27:23.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:27:23.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:27:23.943 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:23.943 INFO:tasks.ceph:role: osd.1 2024-03-25T19:27:23.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:27:23.943 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:27:24.018 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:24.018 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:24.019 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:24.019 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:24.019 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:24.019 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:24.019 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:24.480 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T19:27:24.482 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T19:27:24.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:27:24.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:27:24.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:27:24.666 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:24.667 INFO:tasks.ceph:role: osd.2 2024-03-25T19:27:24.667 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:27:24.667 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:24.741 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:25.233 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T19:27:25.235 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T19:27:25.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:27:25.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:27:25.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:27:25.483 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:25.483 INFO:tasks.ceph:role: osd.3 2024-03-25T19:27:25.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:27:25.483 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:27:25.556 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:25.556 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:25.557 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:26.063 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T19:27:26.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T19:27:26.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:27:26.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:27:26.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:26.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:26.348+0000 7f5a63668640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:27:26.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:26.349+0000 7f5a63668640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:27:26.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:26.349+0000 7f5a63668640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:27:26.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:26.349+0000 7f5a63668640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:27:26.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:26.349+0000 7f5a63668640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:27:26.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:26.349+0000 7f5a63668640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:27:29.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:27:29.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:29.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:29.318+0000 7f98f95ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:27:29.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:29.319+0000 7f98f95ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:27:29.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:29.319+0000 7f98f95ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:27:29.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:29.319+0000 7f98f95ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:27:29.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:29.319+0000 7f98f95ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:27:29.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:29.319+0000 7f98f95ac640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:27:32.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:27:32.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:32.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:32.280+0000 7f25eca11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:27:32.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:32.280+0000 7f25eca11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:27:32.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:32.280+0000 7f25eca11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:27:32.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:32.280+0000 7f25eca11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:27:32.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:32.280+0000 7f25eca11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:27:32.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:32.281+0000 7f25eca11640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:27:35.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:27:35.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:35.242+0000 7f7d21a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:35.242+0000 7f7d21a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:35.242+0000 7f7d21a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:35.242+0000 7f7d21a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:35.242+0000 7f7d21a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:35.243+0000 7f7d21a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:27:38.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:27:38.111 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:27:38.144 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:38.144 INFO:tasks.ceph:role: osd.4 2024-03-25T19:27:38.144 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:27:38.144 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:27:38.221 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:38.221 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:38.222 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:38.738 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T19:27:38.740 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T19:27:38.740 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:27:38.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:27:38.931 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:27:39.001 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:39.001 INFO:tasks.ceph:role: osd.5 2024-03-25T19:27:39.002 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:27:39.002 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:27:39.077 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:39.077 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:39.078 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:39.519 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T19:27:39.521 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T19:27:39.521 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:27:39.619 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:27:39.653 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:27:39.722 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:39.722 INFO:tasks.ceph:role: osd.6 2024-03-25T19:27:39.722 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:27:39.722 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:39.798 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:40.282 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T19:27:40.284 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T19:27:40.284 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:27:40.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:27:40.458 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:27:40.527 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:27:40.527 INFO:tasks.ceph:role: osd.7 2024-03-25T19:27:40.527 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:27:40.527 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:27:40.602 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:27:40.602 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:27:40.602 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:27:40.602 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:27:40.603 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:27:40.603 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:27:40.603 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:27:40.603 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:27:40.603 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:27:40.603 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:27:41.125 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T19:27:41.127 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T19:27:41.128 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:27:41.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:27:41.324 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:41.434 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:41.434+0000 7f14d7ab5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:27:41.434 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:41.434+0000 7f14d7ab5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:27:41.435 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:41.434+0000 7f14d7ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:27:41.435 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:41.434+0000 7f14d7ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:27:41.435 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:41.434+0000 7f14d7ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:27:41.435 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:41.434+0000 7f14d7ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:27:44.282 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:27:44.314 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:44.404 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:44.404+0000 7f4df2bac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:27:44.405 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:44.404+0000 7f4df2bac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:27:44.405 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:44.404+0000 7f4df2bac640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:27:44.405 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:44.404+0000 7f4df2bac640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:27:44.405 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:44.404+0000 7f4df2bac640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:27:44.405 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:44.405+0000 7f4df2bac640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:27:47.252 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:27:47.283 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:47.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:47.374+0000 7fbf9fd73640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:27:47.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:47.375+0000 7fbf9fd73640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:27:47.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:47.375+0000 7fbf9fd73640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:27:47.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:47.375+0000 7fbf9fd73640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:27:47.375 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:47.375+0000 7fbf9fd73640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:27:47.376 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:47.375+0000 7fbf9fd73640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:27:50.229 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:27:50.260 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:50.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:50.351+0000 7fbc8de1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:27:50.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:50.351+0000 7fbc8de1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:27:50.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:50.351+0000 7fbc8de1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:27:50.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:50.351+0000 7fbc8de1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:27:50.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:50.351+0000 7fbc8de1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:27:50.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:27:50.352+0000 7fbc8de1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:27:53.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:27:53.219 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:27:53.220 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:53.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:27:53.251 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:53.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:27:53.322 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:53.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:27:53.392 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:53.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:27:53.462 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:53.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:27:53.530 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:53.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:27:53.562 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:53.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:27:53.632 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:53.632 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:27:53.701 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:53.701 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:27:53.770 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:53.770 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:27:53.838 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:27:53.838 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:27:53.856 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:27:53.856 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:27:53.893 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:27:53.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:27:53.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.y] 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAa0AFmsne6NRAA8tvlfrAVEXGI8BZy5sCkMQ== 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAe0AFmBqzbFBAApdUFmwibqvY5aUV4EpQ1Pg== 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAh0AFmjaASExAAuO7IRHDA2dlYS2fBalkYyg== 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAk0AFmKLPJEBAAdnXESxsuy+oYRJkPX3quPg== 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.3] 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAn0AFmdx6GDhAArN+SWzly+ED3CueGR+U6Gw== 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-03-25T19:27:53.936 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAa0AFm4yBhORAA5YKryeWidwei47n8mivV6Q== 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.4] 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAt0AFmK3L2GRAA8MDSPsDy6kBxhF21LFmW4Q== 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.5] 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAw0AFmmcYxGBAAnnKXvejPoSK3GCr7I+iF8g== 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.6] 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAz0AFmZVtvFhAAx20MoQqx5H37TzOodkEVcg== 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.7] 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA20AFmtHD9FBAAKqo1BBHQMErk64SvFToUkw== 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAb0AFmPzdrABAAy0vXj4d3P5hTZroPpqNCgA== 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout:[client.1] 2024-03-25T19:27:53.937 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAb0AFm/iTqAxAAprJUcPpPx4ghj74G2u85BA== 2024-03-25T19:27:53.958 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.y] 2024-03-25T19:27:53.958 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAa0AFmsne6NRAA8tvlfrAVEXGI8BZy5sCkMQ== 2024-03-25T19:27:53.958 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAe0AFmBqzbFBAApdUFmwibqvY5aUV4EpQ1Pg== 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAh0AFmjaASExAAuO7IRHDA2dlYS2fBalkYyg== 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAk0AFmKLPJEBAAdnXESxsuy+oYRJkPX3quPg== 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.3] 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAn0AFmdx6GDhAArN+SWzly+ED3CueGR+U6Gw== 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAa0AFm4yBhORAA5YKryeWidwei47n8mivV6Q== 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.4] 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAt0AFmK3L2GRAA8MDSPsDy6kBxhF21LFmW4Q== 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.5] 2024-03-25T19:27:53.959 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAw0AFmmcYxGBAAnnKXvejPoSK3GCr7I+iF8g== 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.6] 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAz0AFmZVtvFhAAx20MoQqx5H37TzOodkEVcg== 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.7] 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA20AFmtHD9FBAAKqo1BBHQMErk64SvFToUkw== 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAb0AFmPzdrABAAy0vXj4d3P5hTZroPpqNCgA== 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout:[client.1] 2024-03-25T19:27:53.960 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQAb0AFm/iTqAxAAprJUcPpPx4ghj74G2u85BA== 2024-03-25T19:27:53.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:53.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:54.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.056 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.110 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.165 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.219 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:54.272 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:54.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.325 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.487 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:54.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:54.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:54.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:54.597 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:54.652 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:27:54.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:27:54.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:27:54.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:27:54.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:27:54.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:27:54.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:27:54.994 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:27:55.022 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:27:55.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:27:55.149 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:55.152 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:55.205 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:27:55.205 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:27:55.205 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:27:55.206 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:27:55.208 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:27:55.208 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:27:55.208 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:27:55.208 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:27:55.210 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:27:55.210 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:27:55.210 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:27:55.210 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:27:55.247 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:27:55.247 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:27:55.247 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:27:55.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:27:55.250 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:27:55.250 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:27:55.250 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:27:55.252 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:27:55.252 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:27:55.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.354+0000 7f6bf4b96640 10 monclient: get_monmap_and_config 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.354+0000 7f6bf4b96640 10 monclient: build_initial_monmap 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.354+0000 7f6bf4b96640 10 monclient: monmap: 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:55.355398+0000 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:55.355398+0000 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:27:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:27:55.355 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.354+0000 7f6bf4b96640 1 Processor -- start 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.354+0000 7f6bf4b96640 1 -- start start 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.354+0000 7f6bf4b96640 10 monclient: init 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf00a9920 addr v1:172.21.15.159:6789/0 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf00b45a0 addr v1:172.21.15.134:6790/0 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf0146450 addr v1:172.21.15.134:6789/0 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf006b100 con 0x7f6bf0146450 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00b2940 con 0x7f6bf00a9920 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf013fec0 con 0x7f6bf00b45a0 2024-03-25T19:27:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:55.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bf4b96640 10 monclient(hunting): authenticate will time out at 5907.514379s 2024-03-25T19:27:55.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bf0145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:55.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bf014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:55.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.355+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bf013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:27:55.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.556+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bf014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:55.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.556+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bf0145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:55.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.556+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bf013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:27:55.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.957+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bf014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:55.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.957+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bf0145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:55.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:55.957+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bf013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:27:56.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:56.758+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bf014dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:56.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:56.758+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bf0145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:56.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:56.758+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bf013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bf0145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bf013f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bf014dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf00a9920 addr v1:172.21.15.159:6789/0 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf00b45a0 addr v1:172.21.15.134:6790/0 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf0146450 addr v1:172.21.15.134:6789/0 2024-03-25T19:27:58.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf006b100 con 0x7f6bf0146450 2024-03-25T19:27:58.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00b2940 con 0x7f6bf00a9920 2024-03-25T19:27:58.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf013fec0 con 0x7f6bf00b45a0 2024-03-25T19:27:58.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:58.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:58.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:27:58.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.355+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:58.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.556+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:58.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.556+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:58.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.556+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:27:58.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.957+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:58.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.957+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:58.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:58.957+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:27:59.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:59.758+0000 7f6bedd9b640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:27:59.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:59.758+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:27:59.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:27:59.758+0000 7f6bee59c640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:28:01.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.355+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:01.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.355+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:01.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.355+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.355+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.355+0000 7f6bdffff640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd0008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf00b45a0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf00a9920 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf0146450 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0010e20 con 0x7f6bf0146450 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf006b100 con 0x7f6bf00a9920 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00b2940 con 0x7f6bf00b45a0 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bee59c640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:28:01.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.356+0000 7f6bedd9b640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:28:01.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.557+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:28:01.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.557+0000 7f6bee59c640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:28:01.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.557+0000 7f6bedd9b640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:28:01.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.958+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:28:01.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.958+0000 7f6bee59c640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:28:01.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:01.958+0000 7f6bedd9b640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:28:02.212 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:00:06.859 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:02.404 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:00:07.076 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:02.408 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:00:07.080 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:02.518 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:00:07.164 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:02.712 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:00:07.384 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:02.716 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:00:07.389 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:02.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:02.759+0000 7f6beed9d640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T19:28:02.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:02.759+0000 7f6bee59c640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T19:28:02.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:02.759+0000 7f6bedd9b640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf00b45a0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf00a9920 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf0146450 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf013fec0 con 0x7f6bf0146450 2024-03-25T19:28:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0010e20 con 0x7f6bf00a9920 2024-03-25T19:28:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf006b100 con 0x7f6bf00b45a0 2024-03-25T19:28:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.356+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.396+0000 7f6bee59c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 0x7f6bd000fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:45202/0 (socket says 172.21.15.134:45202) 2024-03-25T19:28:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:04.396+0000 7f6bee59c640 1 -- 172.21.15.134:0/3089356122 learned_addr learned my addr 172.21.15.134:0/3089356122 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:07.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd0010540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6789/0 conn(0x7f6bf0146450 legacy=0x7f6bd000c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf00b45a0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf00a9920 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd0008ef0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00b2940 con 0x7f6bd0008ef0 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0009400 con 0x7f6bf00a9920 2024-03-25T19:28:07.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000a6b0 con 0x7f6bf00b45a0 2024-03-25T19:28:07.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:07.356+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:10.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.159:6789/0 conn(0x7f6bf00a9920 legacy=0x7f6bd0010540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6789/0 conn(0x7f6bd0008ef0 legacy=0x7f6bd00056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd000ce30 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd0010540 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf00b45a0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000b8f0 con 0x7f6bd0010540 2024-03-25T19:28:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000fc10 con 0x7f6bd000ce30 2024-03-25T19:28:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000cb30 con 0x7f6bf00b45a0 2024-03-25T19:28:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:10.356+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6790/0 conn(0x7f6bf00b45a0 legacy=0x7f6bd000fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.159:6789/0 conn(0x7f6bd000ce30 legacy=0x7f6bd000d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6789/0 conn(0x7f6bd0010540 legacy=0x7f6bd000a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd000af50 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd000b390 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd000fe90 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:13.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0010920 con 0x7f6bd000af50 2024-03-25T19:28:13.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000a840 con 0x7f6bd000fe90 2024-03-25T19:28:13.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000a9d0 con 0x7f6bd000b390 2024-03-25T19:28:13.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:13.357+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:16.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:16.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:16.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:16.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:16.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6790/0 conn(0x7f6bd000b390 legacy=0x7f6bd00160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.159:6789/0 conn(0x7f6bd000fe90 legacy=0x7f6bd0011090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6789/0 conn(0x7f6bd000af50 legacy=0x7f6bd00056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd000d210 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd000ebc0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd000f6d0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000abb0 con 0x7f6bd000f6d0 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000d5f0 con 0x7f6bd000ebc0 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000d780 con 0x7f6bd000d210 2024-03-25T19:28:16.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:16.357+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:19.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:19.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:19.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6790/0 conn(0x7f6bd000d210 legacy=0x7f6bd00137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.159:6789/0 conn(0x7f6bd000ebc0 legacy=0x7f6bd000efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6789/0 conn(0x7f6bd000f6d0 legacy=0x7f6bd001b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd0011200 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd000ebc0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd000d210 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0011070 con 0x7f6bd0011200 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000efa0 con 0x7f6bd000ebc0 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000f180 con 0x7f6bd000d210 2024-03-25T19:28:19.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:19.357+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.357+0000 7f6bdffff640 10 monclient(hunting): tick 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.357+0000 7f6bdffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.357+0000 7f6bdffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.357+0000 7f6bdffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6790/0 conn(0x7f6bd000d210 legacy=0x7f6bd00160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.159:6789/0 conn(0x7f6bd000ebc0 legacy=0x7f6bd00056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.357+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6789/0 conn(0x7f6bd0011200 legacy=0x7f6bd00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd0009ce0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd000d210 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd000ebc0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000f3b0 con 0x7f6bd0009ce0 2024-03-25T19:28:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000f540 con 0x7f6bd000ebc0 2024-03-25T19:28:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd000a0c0 con 0x7f6bd000d210 2024-03-25T19:28:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:22.358+0000 7f6bdffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:23.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.282+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911063125 0 0) 0x7f6bd000a0c0 con 0x7f6bd000d210 2024-03-25T19:28:23.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.282+0000 7f6bed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.282+0000 7f6bed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.282+0000 7f6bed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.282+0000 7f6bed59a640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:28:23.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.282+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd8003520 con 0x7f6bd000d210 2024-03-25T19:28:23.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.288+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257918764 0 0) 0x7f6bd000f540 con 0x7f6bd000ebc0 2024-03-25T19:28:23.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.288+0000 7f6bed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.288+0000 7f6bed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.288+0000 7f6bed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.288+0000 7f6bed59a640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:28:23.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.288+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd000a0c0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.297+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306863856 0 0) 0x7f6bd000f3b0 con 0x7f6bd0009ce0 2024-03-25T19:28:23.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.297+0000 7f6bed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.297+0000 7f6bed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.297+0000 7f6bed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.297+0000 7f6bed59a640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:28:23.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.297+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd000f540 con 0x7f6bd0009ce0 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 534371763 0 0) 0x7f6bd000f540 con 0x7f6bd0009ce0 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6790/0 conn(0x7f6bd000d210 legacy=0x7f6bd00160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.159:6789/0 conn(0x7f6bd000ebc0 legacy=0x7f6bd00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 10 monclient: _finish_hunting 0 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 1 monclient: found mon.noname-a 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd000a250 con 0x7f6bd0009ce0 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 10 monclient: _finish_auth 0 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 10 monclient: _check_auth_tickets 2024-03-25T19:28:23.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6be0004080 con 0x7f6bd0009ce0 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bf4b96640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bf4b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bed59a640 10 monclient: dump: 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:23.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.343+0000 7f6bf4b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:23.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6be00041e0 con 0x7f6bd0009ce0 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bed59a640 1 -- 172.21.15.134:0/3089356122 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6be0004630 con 0x7f6bd0009ce0 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bed59a640 10 monclient: dump: 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:23.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bf4b96640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.348+0000 7f6bf4b96640 4 set_mon_vals no callback set 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.349+0000 7f6bf4b96640 10 monclient: shutdown 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.349+0000 7f6bf4b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.349+0000 7f6bf4b96640 1 -- 172.21.15.134:0/3089356122 >> v1:172.21.15.134:6789/0 conn(0x7f6bd0009ce0 legacy=0x7f6bd0013690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.349+0000 7f6bf4b96640 1 -- 172.21.15.134:0/3089356122 shutdown_connections 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.349+0000 7f6bf4b96640 1 -- 172.21.15.134:0/3089356122 >> 172.21.15.134:0/3089356122 conn(0x7f6bf00b7f40 msgr2=0x7f6bf00b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.349+0000 7f6bf4b96640 1 -- 172.21.15.134:0/3089356122 shutdown_connections 2024-03-25T19:28:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.349+0000 7f6bf4b96640 1 -- 172.21.15.134:0/3089356122 wait complete. 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 10 monclient: build_initial_monmap 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 10 monclient: monmap: 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:23.352 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 1 librados: starting msgr at 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 1 librados: starting objecter 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 1 Processor -- start 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 1 -- start start 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 1 librados: setting wanted keys 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 1 librados: calling monclient init 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.350+0000 7f6bf4b96640 10 monclient: init 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd0009ce0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd000d210 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd000ebc0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf01d71a0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf01d7480 con 0x7f6bd000d210 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf01d8020 con 0x7f6bd0009ce0 2024-03-25T19:28:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:23.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.351+0000 7f6bf4b96640 10 monclient(hunting): authenticate will time out at 5935.510212s 2024-03-25T19:28:23.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.352+0000 7f6bedd9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f6bd000d210 0x7f6bf00abb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52734/0 (socket says 172.21.15.134:52734) 2024-03-25T19:28:23.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.352+0000 7f6bedd9b640 1 -- 172.21.15.134:0/774240599 learned_addr learned my addr 172.21.15.134:0/774240599 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:23.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.352+0000 7f6bdeffd640 10 client.?.objecter ms_handle_connect 0x7f6bd000d210 2024-03-25T19:28:23.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.352+0000 7f6bdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:23.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.355+0000 7f6bdeffd640 10 client.?.objecter ms_handle_connect 0x7f6bd000ebc0 2024-03-25T19:28:23.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.355+0000 7f6bdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:23.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.373+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831836043 0 0) 0x7f6bd000a0c0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.373+0000 7f6bdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.373+0000 7f6bdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.373+0000 7f6bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.373+0000 7f6bdeffd640 10 monclient(hunting): my global_id is 4110 2024-03-25T19:28:23.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.373+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd000a3e0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.377+0000 7f6bdeffd640 10 client.?.objecter ms_handle_connect 0x7f6bd0009ce0 2024-03-25T19:28:23.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.377+0000 7f6bdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:23.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.470+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356259041 0 0) 0x7f6bd000a250 con 0x7f6bd0009ce0 2024-03-25T19:28:23.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.470+0000 7f6bdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.470+0000 7f6bdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.470+0000 7f6bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.470+0000 7f6bdeffd640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:28:23.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.470+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd000a0c0 con 0x7f6bd0009ce0 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.472+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277362629 0 0) 0x7f6bd000a3e0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.472+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bd000a250 con 0x7f6bd000ebc0 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.472+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6be8004500 con 0x7f6bd000ebc0 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.472+0000 7f6bdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.472+0000 7f6bdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.472+0000 7f6bdeffd640 10 monclient(hunting): dump: 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:23.473 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:23.473 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:23.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.472+0000 7f6bdeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 481806087 0 0) 0x7f6bd000a250 con 0x7f6bd000ebc0 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 >> v1:172.21.15.134:6790/0 conn(0x7f6bd0009ce0 legacy=0x7f6bf01d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 >> v1:172.21.15.159:6789/0 conn(0x7f6bd000d210 legacy=0x7f6bf00abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 1 monclient: found mon.a 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:23.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf00ad2c0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 5 monclient: authenticate success, global_id 4110 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 10 monclient: _renew_subs 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf01d71a0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 10 monclient: _renew_subs 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf01d71a0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.480+0000 7f6bf4b96640 1 librados: init done 2024-03-25T19:28:23.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.481+0000 7f6bf4b96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:23.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.481+0000 7f6bf4b96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:23.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.481+0000 7f6bf4b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:23.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.481+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf01d71a0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.645+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6be8004a50 con 0x7f6bd000ebc0 2024-03-25T19:28:23.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.645+0000 7f6bdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:23.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.645+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6be8004d60 con 0x7f6bd000ebc0 2024-03-25T19:28:23.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.645+0000 7f6bdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.645+0000 7f6bdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:23.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 10 monclient: dump: 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6be8005290 con 0x7f6bd000ebc0 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6becd99640 4 set_mon_vals no callback set 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3746931900 0 0) 0x7f6be8006830 con 0x7f6bd000ebc0 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 10 client.4110.objecter ms_dispatch 0x7f6bf0000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:28:23.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:28:23.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:28:23.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T19:28:23.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6be8018e00 con 0x7f6bd000ebc0 2024-03-25T19:28:23.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:23.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.646+0000 7f6bdeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:23.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.713+0000 7f6bf4b96640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:28:23.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.713+0000 7f6bf4b96640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:28:23.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.713+0000 7f6bf4b96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:23.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.713+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6bf01d71a0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.877 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:23.877+0000 7fab1464b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:28:23.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.907+0000 7f6bdeffd640 1 -- 172.21.15.134:0/774240599 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f6be80057c0 con 0x7f6bd000ebc0 2024-03-25T19:28:23.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.907+0000 7f6bdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:28:23.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.907+0000 7f6bdeffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:28:23.908 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 10 librados: watch_flush enter 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 10 librados: watch_flush exit 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 10 client.4110.objecter successfully canceled tick 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 10 monclient: shutdown 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 >> v1:172.21.15.134:6789/0 conn(0x7f6bd000ebc0 legacy=0x7f6bf01e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 shutdown_connections 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 >> 172.21.15.134:0/774240599 conn(0x7f6bf00b7f40 msgr2=0x7f6bf0139510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.908+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 shutdown_connections 2024-03-25T19:28:23.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.909+0000 7f6bf4b96640 1 -- 172.21.15.134:0/774240599 wait complete. 2024-03-25T19:28:23.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:23.909+0000 7f6bf4b96640 1 librados: shutdown 2024-03-25T19:28:23.920 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:28:23.920 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:28:23.920 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:28:23.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:28:23.932 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:23.932+0000 7fab1464b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:28:23.948 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:28:23.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:28:24.016 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:28:24.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:28:24.084 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T19:28:24.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:28:24.151 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:28:24.151 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:28:24.167 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:24.167+0000 7f9760632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:28:24.179 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:28:24.179 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:28:24.227 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:24.227+0000 7f9760632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:28:24.246 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:28:24.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:28:24.313 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T19:28:24.314 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:28:24.380 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 6064f78c-3ecd-4a52-bbbe-608a8fdf3bca 0 2024-03-25T19:28:24.522 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.522+0000 7fe75277c640 10 monclient: get_monmap_and_config 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.522+0000 7fe75277c640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.522+0000 7fe75277c640 10 monclient: monmap: 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:24.524165+0000 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:24.524165+0000 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:24.523 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:24.524 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 1 Processor -- start 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 1 -- start start 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient: init 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient(hunting): picked mon.noname-c con 0x7fe74c0b4700 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient(hunting): picked mon.noname-b con 0x7fe74c142a20 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient(hunting): picked mon.noname-a con 0x7fe74c0a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c0a9d60 con 0x7fe74c0a96b0 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c13ff10 con 0x7fe74c0b4700 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c0aa1f0 con 0x7fe74c142a20 2024-03-25T19:28:24.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.526 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.523+0000 7fe75277c640 10 monclient(hunting): authenticate will time out at 5914.833664s 2024-03-25T19:28:24.526 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.524+0000 7fe750d19640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe74c0a96b0 0x7fe74c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41248/0 (socket says 172.21.15.159:41248) 2024-03-25T19:28:24.526 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.524+0000 7fe750d19640 1 -- 172.21.15.159:0/180681584 learned_addr learned my addr 172.21.15.159:0/180681584 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:24.526 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.526+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880078536 0 0) 0x7fe74c13ff10 con 0x7fe74c0b4700 2024-03-25T19:28:24.526 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.526+0000 7fe74affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.526 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.526+0000 7fe74affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.526 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.526+0000 7fe74affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.526+0000 7fe74affd640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.526+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe734003520 con 0x7fe74c0b4700 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.527+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949568047 0 0) 0x7fe74c0aa1f0 con 0x7fe74c142a20 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.527+0000 7fe74affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.527+0000 7fe74affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.527+0000 7fe74affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.527+0000 7fe74affd640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:28:24.527 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.527+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c13ff10 con 0x7fe74c142a20 2024-03-25T19:28:24.528 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.528+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876075577 0 0) 0x7fe74c0a9d60 con 0x7fe74c0a96b0 2024-03-25T19:28:24.528 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.528+0000 7fe74affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.528 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.528+0000 7fe74affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.529 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.528+0000 7fe74affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.529 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.528+0000 7fe74affd640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:28:24.529 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.528+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c0aa1f0 con 0x7fe74c0a96b0 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4051929615 0 0) 0x7fe74c13ff10 con 0x7fe74c142a20 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 >> v1:172.21.15.159:6789/0 conn(0x7fe74c0b4700 legacy=0x7fe74c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 >> v1:172.21.15.134:6789/0 conn(0x7fe74c0a96b0 legacy=0x7fe74c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 1 monclient: found mon.noname-b 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c0b4ae0 con 0x7fe74c142a20 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe74affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe75277c640 5 monclient: authenticate success, global_id 4115 2024-03-25T19:28:24.531 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.530+0000 7fe75277c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.531+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe740001630 con 0x7fe74c142a20 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.531+0000 7fe74affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.531+0000 7fe74affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.531+0000 7fe74affd640 10 monclient: dump: 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.532 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.533 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.533 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.533 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.533 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.531+0000 7fe75277c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.533+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe740001880 con 0x7fe74c142a20 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe74affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe74affd640 1 -- 172.21.15.159:0/180681584 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe740001c10 con 0x7fe74c142a20 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe74affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe74affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe74affd640 10 monclient: dump: 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.534 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.535 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.535 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.535 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.535 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.535 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.535 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:24.535 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 4 set_mon_vals no callback set 2024-03-25T19:28:24.537 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 10 monclient: shutdown 2024-03-25T19:28:24.537 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:24.537 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 1 -- 172.21.15.159:0/180681584 >> v1:172.21.15.134:6790/0 conn(0x7fe74c142a20 legacy=0x7fe74c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 1 -- 172.21.15.159:0/180681584 shutdown_connections 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 1 -- 172.21.15.159:0/180681584 >> 172.21.15.159:0/180681584 conn(0x7fe74c0668e0 msgr2=0x7fe74c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 1 -- 172.21.15.159:0/180681584 shutdown_connections 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.534+0000 7fe75277c640 1 -- 172.21.15.159:0/180681584 wait complete. 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.535+0000 7fe75277c640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.535+0000 7fe75277c640 10 monclient: monmap: 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:24.538 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.535+0000 7fe75277c640 1 librados: starting msgr at 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.535+0000 7fe75277c640 1 librados: starting objecter 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.535+0000 7fe75277c640 1 Processor -- start 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.535+0000 7fe75277c640 1 -- start start 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.535+0000 7fe75277c640 1 librados: setting wanted keys 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 1 librados: calling monclient init 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient: init 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient(hunting): picked mon.noname-b con 0x7fe74c0a96b0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient(hunting): picked mon.noname-c con 0x7fe74c0b4700 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient(hunting): picked mon.noname-a con 0x7fe74c142a20 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c138960 con 0x7fe74c142a20 2024-03-25T19:28:24.539 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c139bb0 con 0x7fe74c0a96b0 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c13a650 con 0x7fe74c0b4700 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe75277c640 10 monclient(hunting): authenticate will time out at 5914.846267s 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe750d19640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe74c142a20 0x7fe74c1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41256/0 (socket says 172.21.15.159:41256) 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe74b7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe74c0b4700 0x7fe74c139f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50290/0 (socket says 172.21.15.159:50290) 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.536+0000 7fe750d19640 1 -- 172.21.15.159:0/3682939511 learned_addr learned my addr 172.21.15.159:0/3682939511 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.537+0000 7fe748ff9640 10 client.?.objecter ms_handle_connect 0x7fe74c142a20 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.537+0000 7fe748ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.537+0000 7fe748ff9640 10 client.?.objecter ms_handle_connect 0x7fe74c0b4700 2024-03-25T19:28:24.540 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.537+0000 7fe748ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175234738 0 0) 0x7fe74c138960 con 0x7fe74c142a20 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe738004eb0 con 0x7fe74c142a20 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393573959 0 0) 0x7fe74c13a650 con 0x7fe74c0b4700 2024-03-25T19:28:24.541 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.542 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.542 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.542 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:28:24.542 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.539+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c138960 con 0x7fe74c0b4700 2024-03-25T19:28:24.544 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.544+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3364645986 0 0) 0x7fe738004eb0 con 0x7fe74c142a20 2024-03-25T19:28:24.544 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.544+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe74c13a650 con 0x7fe74c142a20 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.544+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe744001c70 con 0x7fe74c142a20 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.544+0000 7fe748ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.544+0000 7fe748ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.544+0000 7fe748ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.545 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.544+0000 7fe748ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:24.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3437603198 0 0) 0x7fe74c13a650 con 0x7fe74c142a20 2024-03-25T19:28:24.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 >> v1:172.21.15.134:6790/0 conn(0x7fe74c0b4700 legacy=0x7fe74c139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 >> v1:172.21.15.159:6789/0 conn(0x7fe74c0a96b0 legacy=0x7fe74c1380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 1 monclient: found mon.a 2024-03-25T19:28:24.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.547 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c13cd60 con 0x7fe74c142a20 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe748ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 10 monclient: _renew_subs 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74c136960 con 0x7fe74c142a20 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 10 monclient: _renew_subs 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c137750 con 0x7fe74c142a20 2024-03-25T19:28:24.548 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fe75277c640 1 librados: init done 2024-03-25T19:28:24.548 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.547+0000 7fab1464b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:28:24.549 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.548+0000 7fe75277c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.549 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.548+0000 7fe75277c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.549 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.548+0000 7fe75277c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.549 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.548+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74c04f730 con 0x7fe74c142a20 2024-03-25T19:28:24.603 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.603+0000 7fab1464b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:28:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe744004b60 con 0x7fe74c142a20 2024-03-25T19:28:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe744004d50 con 0x7fe74c142a20 2024-03-25T19:28:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 10 monclient: dump: 2024-03-25T19:28:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe744005300 con 0x7fe74c142a20 2024-03-25T19:28:24.646 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe74a7fc640 4 set_mon_vals no callback set 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3203523370 0 0) 0x7fe7440067d0 con 0x7fe74c142a20 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 10 client.4119.objecter ms_dispatch 0x7fe74c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.645+0000 7fe748ff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.646+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe7440057e0 con 0x7fe74c142a20 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.646+0000 7fe748ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.646+0000 7fe748ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:24.648 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.647+0000 7fab1464b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:28:24.703 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.703+0000 7fab1464b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:28:24.718 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.718+0000 7fe75277c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6064f78c-3ecd-4a52-bbbe-608a8fdf3bca", "id": 0}] 2024-03-25T19:28:24.718 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.718+0000 7fe75277c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6064f78c-3ecd-4a52-bbbe-608a8fdf3bca", "id": 0}] 2024-03-25T19:28:24.718 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.718+0000 7fe75277c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.719 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.718+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6064f78c-3ecd-4a52-bbbe-608a8fdf3bca", "id": 0} v 0) v1 -- 0x7fe74c0b7030 con 0x7fe74c142a20 2024-03-25T19:28:24.745 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.745+0000 7fab1464b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:28:24.789 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:24.789+0000 7f9760632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:28:24.825 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.825+0000 7fe748ff9640 1 -- 172.21.15.159:0/3682939511 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6064f78c-3ecd-4a52-bbbe-608a8fdf3bca", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3509742566 0 732707515) 0x7fe744005980 con 0x7fe74c142a20 2024-03-25T19:28:24.826 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.825+0000 7fe748ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6064f78c-3ecd-4a52-bbbe-608a8fdf3bca", "id": 0}] 2024-03-25T19:28:24.826 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.825+0000 7fe748ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:24.826 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-03-25T19:28:24.827 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 10 librados: watch_flush enter 2024-03-25T19:28:24.827 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 10 librados: watch_flush exit 2024-03-25T19:28:24.827 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:28:24.827 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:28:24.827 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 10 monclient: shutdown 2024-03-25T19:28:24.827 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:24.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 >> v1:172.21.15.134:6789/0 conn(0x7fe74c142a20 legacy=0x7fe74c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 shutdown_connections 2024-03-25T19:28:24.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 >> 172.21.15.159:0/3682939511 conn(0x7fe74c0668e0 msgr2=0x7fe74c13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:24.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 shutdown_connections 2024-03-25T19:28:24.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 1 -- 172.21.15.159:0/3682939511 wait complete. 2024-03-25T19:28:24.828 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.826+0000 7fe75277c640 1 librados: shutdown 2024-03-25T19:28:24.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 9c95d080-d8a1-4d86-afea-ed5e71b09634 1 2024-03-25T19:28:24.847 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:24.847+0000 7f9760632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:28:24.866 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.866+0000 7fab1464b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:28:24.893 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:24.894+0000 7f9760632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:28:24.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.936+0000 7f23914f1640 10 monclient: get_monmap_and_config 2024-03-25T19:28:24.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.936+0000 7f23914f1640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.936+0000 7f23914f1640 10 monclient: monmap: 2024-03-25T19:28:24.936 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:24.936 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.936 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:24.937803+0000 2024-03-25T19:28:24.936 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:24.937803+0000 2024-03-25T19:28:24.937 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.937 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.937 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:24.937 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:24.937 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:24.937 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.936+0000 7f23914f1640 1 Processor -- start 2024-03-25T19:28:24.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.936+0000 7f23914f1640 1 -- start start 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.936+0000 7f23914f1640 10 monclient: init 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 10 monclient(hunting): picked mon.noname-a con 0x7f238c06c2b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 10 monclient(hunting): picked mon.noname-c con 0x7f238c0b0100 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 10 monclient(hunting): picked mon.noname-b con 0x7f238c146300 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c13feb0 con 0x7f238c06c2b0 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c06d050 con 0x7f238c0b0100 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0b06d0 con 0x7f238c146300 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.937+0000 7f23914f1640 10 monclient(hunting): authenticate will time out at 5915.247164s 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.938+0000 7f238affd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f238c06c2b0 0x7f238c0af9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41268/0 (socket says 172.21.15.159:41268) 2024-03-25T19:28:24.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.938+0000 7f238affd640 1 -- 172.21.15.159:0/3982788549 learned_addr learned my addr 172.21.15.159:0/3982788549 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:24.940 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.938+0000 7f238b7fe640 1 --1- 172.21.15.159:0/3982788549 >> v1:172.21.15.134:6790/0 conn(0x7f238c146300 0x7f238c14db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50296/0 (socket says 172.21.15.159:50296) 2024-03-25T19:28:24.940 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.938+0000 7fab1464b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284794859 0 0) 0x7f238c13feb0 con 0x7f238c06c2b0 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2374003520 con 0x7f238c06c2b0 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991891463 0 0) 0x7f238c0b06d0 con 0x7f238c146300 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c13feb0 con 0x7f238c146300 2024-03-25T19:28:24.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020188992 0 0) 0x7f238c06d050 con 0x7f238c0b0100 2024-03-25T19:28:24.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:28:24.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.940+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c0b06d0 con 0x7f238c0b0100 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3362400283 0 0) 0x7f2374003520 con 0x7f238c06c2b0 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 >> v1:172.21.15.134:6790/0 conn(0x7f238c146300 legacy=0x7f238c14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 >> v1:172.21.15.159:6789/0 conn(0x7f238c0b0100 legacy=0x7f238c145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 1 monclient: found mon.noname-a 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c06d4a0 con 0x7f238c06c2b0 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.943+0000 7f2389ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f2389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f237c001690 con 0x7f238c06c2b0 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f23914f1640 5 monclient: authenticate success, global_id 4122 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f23914f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f2389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f2389ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f2389ffb640 10 monclient: dump: 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.945 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.946 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.946 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.946 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.946 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.946 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.946 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.944+0000 7f23914f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:24.947 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f237c0017f0 con 0x7f238c06c2b0 2024-03-25T19:28:24.947 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f2389ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f2389ffb640 1 -- 172.21.15.159:0/3982788549 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f237c001c40 con 0x7f238c06c2b0 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f2389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f2389ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f2389ffb640 10 monclient: dump: 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.948 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f23914f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.945+0000 7f23914f1640 4 set_mon_vals no callback set 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.946+0000 7f23914f1640 10 monclient: shutdown 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.946+0000 7f23914f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.946+0000 7f23914f1640 1 -- 172.21.15.159:0/3982788549 >> v1:172.21.15.134:6789/0 conn(0x7f238c06c2b0 legacy=0x7f238c0af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.946+0000 7f23914f1640 1 -- 172.21.15.159:0/3982788549 shutdown_connections 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.946+0000 7f23914f1640 1 -- 172.21.15.159:0/3982788549 >> 172.21.15.159:0/3982788549 conn(0x7f238c0b5cd0 msgr2=0x7f238c0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.946+0000 7f23914f1640 1 -- 172.21.15.159:0/3982788549 shutdown_connections 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.946+0000 7f23914f1640 1 -- 172.21.15.159:0/3982788549 wait complete. 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 10 monclient: monmap: 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:24.949 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 1 librados: starting msgr at 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 1 librados: starting objecter 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 1 Processor -- start 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 1 -- start start 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 1 librados: setting wanted keys 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 1 librados: calling monclient init 2024-03-25T19:28:24.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.947+0000 7f23914f1640 10 monclient: init 2024-03-25T19:28:24.951 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:24.948+0000 7f9760632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 10 monclient(hunting): picked mon.noname-c con 0x7f238c06c2b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 10 monclient(hunting): picked mon.noname-a con 0x7f238c0b0100 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 10 monclient(hunting): picked mon.noname-b con 0x7f238c146300 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c142770 con 0x7f238c0b0100 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c071040 con 0x7f238c146300 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0711d0 con 0x7f238c06c2b0 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.953 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f23914f1640 10 monclient(hunting): authenticate will time out at 5915.258204s 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f238a7fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f238c0b0100 0x7f238c1da8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41274/0 (socket says 172.21.15.159:41274) 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.948+0000 7f238a7fc640 1 -- 172.21.15.159:0/1148942226 learned_addr learned my addr 172.21.15.159:0/1148942226 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.949+0000 7f23737fe640 10 client.?.objecter ms_handle_connect 0x7f238c0b0100 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.949+0000 7f23737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.949+0000 7f23737fe640 10 client.?.objecter ms_handle_connect 0x7f238c06c2b0 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.949+0000 7f23737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 client.?.objecter ms_handle_connect 0x7f238c146300 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018062498 0 0) 0x7f238c142770 con 0x7f238c0b0100 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.954 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2378004eb0 con 0x7f238c0b0100 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658064598 0 0) 0x7f238c0711d0 con 0x7f238c06c2b0 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.951+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c142770 con 0x7f238c06c2b0 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.953+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410242433 0 0) 0x7f238c071040 con 0x7f238c146300 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.953+0000 7f23737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.953+0000 7f23737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.953+0000 7f23737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.953+0000 7f23737fe640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:28:24.955 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.953+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c0711d0 con 0x7f238c146300 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036531472 0 0) 0x7f238c142770 con 0x7f238c06c2b0 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f238c071040 con 0x7f238c06c2b0 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2752720156 0 0) 0x7f2378004eb0 con 0x7f238c0b0100 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f238c142770 con 0x7f238c0b0100 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2380001a10 con 0x7f238c0b0100 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 10 monclient(hunting): dump: 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.957 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.958 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.956+0000 7f23737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:24.961 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1600405035 0 0) 0x7f238c142770 con 0x7f238c0b0100 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 >> v1:172.21.15.134:6790/0 conn(0x7f238c06c2b0 legacy=0x7f238c13ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 >> v1:172.21.15.159:6789/0 conn(0x7f238c146300 legacy=0x7f238c1e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 1 monclient: found mon.a 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c071360 con 0x7f238c0b0100 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 10 monclient: _renew_subs 2024-03-25T19:28:24.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c1420c0 con 0x7f238c0b0100 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 10 monclient: _renew_subs 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c13b4a0 con 0x7f238c0b0100 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.961+0000 7f23914f1640 1 librados: init done 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.962+0000 7f23914f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.963 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.962+0000 7f23914f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.964 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.962+0000 7f23914f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:24.964 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.962+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c04f730 con 0x7f238c0b0100 2024-03-25T19:28:24.966 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2380001c70 con 0x7f238c0b0100 2024-03-25T19:28:24.966 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.966 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2380003a60 con 0x7f238c0b0100 2024-03-25T19:28:24.966 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 10 monclient: dump: 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2380004010 con 0x7f238c0b0100 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23897fa640 4 set_mon_vals no callback set 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 815593883 0 0) 0x7f2380006670 con 0x7f238c0b0100 2024-03-25T19:28:24.967 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 10 client.4125.objecter ms_dispatch 0x7f238c000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:28:24.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:28:24.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:28:24.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:24.966+0000 7f23737fe640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:28:24.988 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:24.988+0000 7fab1464b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:28:24.993 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:24.993+0000 7f9760632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:28:25.040 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:25.040+0000 7fab1464b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:28:25.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.062+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f23800042c0 con 0x7f238c0b0100 2024-03-25T19:28:25.063 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.062+0000 7f23737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.063 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.062+0000 7f23737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:25.089 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:25.089+0000 7fab1464b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:28:25.112 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:25.112+0000 7f9760632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:28:25.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.134+0000 7f23914f1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9c95d080-d8a1-4d86-afea-ed5e71b09634", "id": 1}] 2024-03-25T19:28:25.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.134+0000 7f23914f1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9c95d080-d8a1-4d86-afea-ed5e71b09634", "id": 1}] 2024-03-25T19:28:25.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.134+0000 7f23914f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.134+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9c95d080-d8a1-4d86-afea-ed5e71b09634", "id": 1} v 0) v1 -- 0x7f238c0adda0 con 0x7f238c0b0100 2024-03-25T19:28:25.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.172+0000 7f23737fe640 1 -- 172.21.15.159:0/1148942226 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c95d080-d8a1-4d86-afea-ed5e71b09634", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4202208989 0 940483276) 0x7f2380006930 con 0x7f238c0b0100 2024-03-25T19:28:25.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.172+0000 7f23737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9c95d080-d8a1-4d86-afea-ed5e71b09634", "id": 1}] 2024-03-25T19:28:25.172 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.172+0000 7f23737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:25.172 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-03-25T19:28:25.173 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 10 librados: watch_flush enter 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 10 librados: watch_flush exit 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 10 monclient: shutdown 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 >> v1:172.21.15.134:6789/0 conn(0x7f238c0b0100 legacy=0x7f238c1da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 shutdown_connections 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 >> 172.21.15.159:0/1148942226 conn(0x7f238c0b5cd0 msgr2=0x7f238c0b35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 shutdown_connections 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 1 -- 172.21.15.159:0/1148942226 wait complete. 2024-03-25T19:28:25.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.173+0000 7f23914f1640 1 librados: shutdown 2024-03-25T19:28:25.185 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:25.186+0000 7f9760632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:25.187 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9 2 2024-03-25T19:28:25.234 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:25.235+0000 7f9760632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:28:25.286 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:25.286+0000 7f9760632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.326+0000 7fa5b4806640 10 monclient: get_monmap_and_config 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.326+0000 7fa5b4806640 10 monclient: build_initial_monmap 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.326+0000 7fa5b4806640 10 monclient: monmap: 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:25.328098+0000 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:25.328098+0000 2024-03-25T19:28:25.327 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 1 Processor -- start 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 1 -- start start 2024-03-25T19:28:25.328 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient: init 2024-03-25T19:28:25.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:25.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:25.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b00b4700 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:25.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b0142a20 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:25.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b00a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:25.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00a9d60 con 0x7fa5b00a96b0 2024-03-25T19:28:25.329 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b013ff10 con 0x7fa5b0142a20 2024-03-25T19:28:25.330 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00aa1f0 con 0x7fa5b00b4700 2024-03-25T19:28:25.330 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:25.330 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.327+0000 7fa5b4806640 10 monclient(hunting): authenticate will time out at 5915.637619s 2024-03-25T19:28:25.330 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.328+0000 7fa5ae59c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fa5b00b4700 0x7fa5b0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50314/0 (socket says 172.21.15.159:50314) 2024-03-25T19:28:25.330 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.328+0000 7fa5ae59c640 1 -- 172.21.15.159:0/3974156767 learned_addr learned my addr 172.21.15.159:0/3974156767 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:25.330 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.328+0000 7fa5aed9d640 1 --1- 172.21.15.159:0/3974156767 >> v1:172.21.15.134:6789/0 conn(0x7fa5b00a96b0 0x7fa5b013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41290/0 (socket says 172.21.15.159:41290) 2024-03-25T19:28:25.330 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.330+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494953068 0 0) 0x7fa5b00a9d60 con 0x7fa5b00a96b0 2024-03-25T19:28:25.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.330+0000 7fa5ad59a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.330+0000 7fa5ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.330+0000 7fa5ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.330+0000 7fa5ad59a640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:28:25.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.330+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598003520 con 0x7fa5b00a96b0 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821612746 0 0) 0x7fa5b013ff10 con 0x7fa5b0142a20 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b00a9d60 con 0x7fa5b0142a20 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473914702 0 0) 0x7fa5b00aa1f0 con 0x7fa5b00b4700 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:28:25.332 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.331+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b013ff10 con 0x7fa5b00b4700 2024-03-25T19:28:25.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1037641496 0 0) 0x7fa598003520 con 0x7fa5b00a96b0 2024-03-25T19:28:25.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 >> v1:172.21.15.134:6790/0 conn(0x7fa5b00b4700 legacy=0x7fa5b0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 >> v1:172.21.15.159:6789/0 conn(0x7fa5b0142a20 legacy=0x7fa5b013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 10 monclient: _finish_hunting 0 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 1 monclient: found mon.noname-a 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b00b4ae0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 10 monclient: _finish_auth 0 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 10 monclient: _check_auth_tickets 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa5a8001a10 con 0x7fa5b00a96b0 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5ad59a640 10 monclient: dump: 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5b4806640 5 monclient: authenticate success, global_id 4128 2024-03-25T19:28:25.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.334+0000 7fa5b4806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:25.337 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:25.335+0000 7f9760632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:28:25.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a8001b70 con 0x7fa5b00a96b0 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5ad59a640 1 -- 172.21.15.159:0/3974156767 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa5a8001f80 con 0x7fa5b00a96b0 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5b4806640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5b4806640 4 set_mon_vals no callback set 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.335+0000 7fa5ad59a640 10 monclient: dump: 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.340 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.336+0000 7fa5b4806640 10 monclient: shutdown 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.336+0000 7fa5b4806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.336+0000 7fa5b4806640 1 -- 172.21.15.159:0/3974156767 >> v1:172.21.15.134:6789/0 conn(0x7fa5b00a96b0 legacy=0x7fa5b013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.336+0000 7fa5b4806640 1 -- 172.21.15.159:0/3974156767 shutdown_connections 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.336+0000 7fa5b4806640 1 -- 172.21.15.159:0/3974156767 >> 172.21.15.159:0/3974156767 conn(0x7fa5b00668e0 msgr2=0x7fa5b0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.336+0000 7fa5b4806640 1 -- 172.21.15.159:0/3974156767 shutdown_connections 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.336+0000 7fa5b4806640 1 -- 172.21.15.159:0/3974156767 wait complete. 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.337+0000 7fa5b4806640 10 monclient: build_initial_monmap 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.337+0000 7fa5b4806640 10 monclient: monmap: 2024-03-25T19:28:25.341 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.337+0000 7fa5b4806640 1 librados: starting msgr at 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.337+0000 7fa5b4806640 1 librados: starting objecter 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.337+0000 7fa5b4806640 1 Processor -- start 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 1 -- start start 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 1 librados: setting wanted keys 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 1 librados: calling monclient init 2024-03-25T19:28:25.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient: init 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b00a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b00b4700 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b0142a20 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00aba20 con 0x7fa5b00a96b0 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00abd00 con 0x7fa5b00b4700 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b013bd80 con 0x7fa5b0142a20 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5b4806640 10 monclient(hunting): authenticate will time out at 5915.648266s 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5ae59c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa5b00a96b0 0x7fa5b00ac120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41302/0 (socket says 172.21.15.159:41302) 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.338+0000 7fa5ae59c640 1 -- 172.21.15.159:0/3878268107 learned_addr learned my addr 172.21.15.159:0/3878268107 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:25.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.339+0000 7fa59effd640 10 client.?.objecter ms_handle_connect 0x7fa5b00a96b0 2024-03-25T19:28:25.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.339+0000 7fa59effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:25.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.339+0000 7fa59effd640 10 client.?.objecter ms_handle_connect 0x7fa5b0142a20 2024-03-25T19:28:25.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.339+0000 7fa59effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:25.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.339+0000 7fa59effd640 10 client.?.objecter ms_handle_connect 0x7fa5b00b4700 2024-03-25T19:28:25.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.339+0000 7fa59effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075597653 0 0) 0x7fa5b00aba20 con 0x7fa5b00a96b0 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa590004eb0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965656413 0 0) 0x7fa5b013bd80 con 0x7fa5b0142a20 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.341+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b00aba20 con 0x7fa5b0142a20 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.342+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510110151 0 0) 0x7fa5b00abd00 con 0x7fa5b00b4700 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.342+0000 7fa59effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.342+0000 7fa59effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.342+0000 7fa59effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.342+0000 7fa59effd640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:28:25.346 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.342+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b013bd80 con 0x7fa5b00b4700 2024-03-25T19:28:25.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.345+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 690810772 0 0) 0x7fa590004eb0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.345+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b00abd00 con 0x7fa5b00a96b0 2024-03-25T19:28:25.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.345+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa5a0001910 con 0x7fa5b00a96b0 2024-03-25T19:28:25.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.345+0000 7fa59effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.345+0000 7fa59effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:25.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.345+0000 7fa59effd640 10 monclient(hunting): dump: 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.345+0000 7fa59effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.347+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 674096134 0 0) 0x7fa5b013bd80 con 0x7fa5b00b4700 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.347+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa590004eb0 con 0x7fa5b00b4700 2024-03-25T19:28:25.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.347+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa5a40018f0 con 0x7fa5b00b4700 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.347+0000 7fa59effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.347+0000 7fa59effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.347+0000 7fa59effd640 10 monclient(hunting): dump: 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.349 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3292840545 0 0) 0x7fa5b00aba20 con 0x7fa5b0142a20 2024-03-25T19:28:25.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b013bd80 con 0x7fa5b0142a20 2024-03-25T19:28:25.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa5a8001e80 con 0x7fa5b0142a20 2024-03-25T19:28:25.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 10 monclient(hunting): dump: 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3648183345 0 0) 0x7fa5b00abd00 con 0x7fa5b00a96b0 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 >> v1:172.21.15.134:6790/0 conn(0x7fa5b0142a20 legacy=0x7fa5b01e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 >> v1:172.21.15.159:6789/0 conn(0x7fa5b00b4700 legacy=0x7fa5b01392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 monclient: found mon.a 2024-03-25T19:28:25.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b013ccb0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 10 monclient: _finish_auth 0 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa59effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa5b4806640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa5b4806640 10 monclient: _renew_subs 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa5b4806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.349+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b013ac10 con 0x7fa5b00a96b0 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 10 monclient: _renew_subs 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b013b8d0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 1 librados: init done 2024-03-25T19:28:25.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.354 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.350+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b013bbd0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a00047e0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa5a0004bf0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 10 monclient: dump: 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.355 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa5a00051a0 con 0x7fa5b00a96b0 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa5acd99640 4 set_mon_vals no callback set 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2892684500 0 0) 0x7fa5a0006690 con 0x7fa5b00a96b0 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 10 client.4131.objecter ms_dispatch 0x7fa5b0000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:28:25.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.353+0000 7fa59effd640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:28:25.448 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.448+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa5a0006380 con 0x7fa5b00a96b0 2024-03-25T19:28:25.448 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.448+0000 7fa59effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.448 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.448+0000 7fa59effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:25.500 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:25.499+0000 7fab1464b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:28:25.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.525+0000 7fa5b4806640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9", "id": 2}] 2024-03-25T19:28:25.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.525+0000 7fa5b4806640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9", "id": 2}] 2024-03-25T19:28:25.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.525+0000 7fa5b4806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.525 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.525+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9", "id": 2} v 0) v1 -- 0x7fa5b013bf10 con 0x7fa5b00a96b0 2024-03-25T19:28:25.571 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.570+0000 7fa59effd640 1 -- 172.21.15.159:0/3878268107 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3756636123 0 216599125) 0x7fa5a0005520 con 0x7fa5b00a96b0 2024-03-25T19:28:25.571 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.570+0000 7fa59effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9", "id": 2}] 2024-03-25T19:28:25.571 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.570+0000 7fa59effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:25.571 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-03-25T19:28:25.572 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 10 librados: watch_flush enter 2024-03-25T19:28:25.572 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 10 librados: watch_flush exit 2024-03-25T19:28:25.572 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:28:25.572 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:28:25.572 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 10 monclient: shutdown 2024-03-25T19:28:25.573 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:25.573 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 >> v1:172.21.15.134:6789/0 conn(0x7fa5b00a96b0 legacy=0x7fa5b00ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.573 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 shutdown_connections 2024-03-25T19:28:25.573 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.571+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 >> 172.21.15.159:0/3878268107 conn(0x7fa5b00668e0 msgr2=0x7fa5b013d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:25.573 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.572+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 shutdown_connections 2024-03-25T19:28:25.573 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.572+0000 7fa5b4806640 1 -- 172.21.15.159:0/3878268107 wait complete. 2024-03-25T19:28:25.573 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.572+0000 7fa5b4806640 1 librados: shutdown 2024-03-25T19:28:25.582 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new dfd685c1-d3ec-4f5d-9e86-56b1c412f421 3 2024-03-25T19:28:25.642 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:25.642+0000 7fab1464b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:28:25.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.682+0000 7feb8ba63640 10 monclient: get_monmap_and_config 2024-03-25T19:28:25.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.682+0000 7feb8ba63640 10 monclient: build_initial_monmap 2024-03-25T19:28:25.682 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.682+0000 7feb8ba63640 10 monclient: monmap: 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:25.683919+0000 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:25.683919+0000 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:25.683 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.683+0000 7feb8ba63640 1 Processor -- start 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.683+0000 7feb8ba63640 1 -- start start 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.683+0000 7feb8ba63640 10 monclient: init 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 10 monclient(hunting): picked mon.noname-c con 0x7feb840b4700 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 10 monclient(hunting): picked mon.noname-a con 0x7feb84142a20 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:25.685 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 10 monclient(hunting): picked mon.noname-b con 0x7feb840a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb840a9d60 con 0x7feb84142a20 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8413ff10 con 0x7feb840b4700 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb840aa1f0 con 0x7feb840a96b0 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8ba63640 10 monclient(hunting): authenticate will time out at 5915.993868s 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb88ffe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7feb84142a20 0x7feb8413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41308/0 (socket says 172.21.15.159:41308) 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb8a000640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7feb840a96b0 0x7feb8413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50328/0 (socket says 172.21.15.159:50328) 2024-03-25T19:28:25.686 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb88ffe640 1 -- 172.21.15.159:0/1468326515 learned_addr learned my addr 172.21.15.159:0/1468326515 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:25.687 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.684+0000 7feb897ff640 1 --1- 172.21.15.159:0/1468326515 >> v1:172.21.15.159:6789/0 conn(0x7feb840b4700 0x7feb84142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:47326/0 (socket says 172.21.15.159:47326) 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385216603 0 0) 0x7feb840a9d60 con 0x7feb84142a20 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70003520 con 0x7feb84142a20 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188861199 0 0) 0x7feb8413ff10 con 0x7feb840b4700 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb840a9d60 con 0x7feb840b4700 2024-03-25T19:28:25.688 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682223422 0 0) 0x7feb840aa1f0 con 0x7feb840a96b0 2024-03-25T19:28:25.689 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.689 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.689 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.689 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:28:25.689 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.687+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8413ff10 con 0x7feb840a96b0 2024-03-25T19:28:25.690 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264078942 0 0) 0x7feb70003520 con 0x7feb84142a20 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 >> v1:172.21.15.134:6790/0 conn(0x7feb840a96b0 legacy=0x7feb8413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 >> v1:172.21.15.159:6789/0 conn(0x7feb840b4700 legacy=0x7feb84142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 1 monclient: found mon.noname-a 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb840b4ae0 con 0x7feb84142a20 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 10 monclient: _finish_auth 0 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb8ba63640 5 monclient: authenticate success, global_id 4134 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb8ba63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7feb74001690 con 0x7feb84142a20 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb7bfff640 10 monclient: dump: 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.692 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.690+0000 7feb8ba63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:25.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb74001870 con 0x7feb84142a20 2024-03-25T19:28:25.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:25.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb7bfff640 1 -- 172.21.15.159:0/1468326515 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7feb74001c00 con 0x7feb84142a20 2024-03-25T19:28:25.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:25.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 4 set_mon_vals no callback set 2024-03-25T19:28:25.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb7bfff640 10 monclient: dump: 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.696 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 10 monclient: shutdown 2024-03-25T19:28:25.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:25.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 1 -- 172.21.15.159:0/1468326515 >> v1:172.21.15.134:6789/0 conn(0x7feb84142a20 legacy=0x7feb8413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 1 -- 172.21.15.159:0/1468326515 shutdown_connections 2024-03-25T19:28:25.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 1 -- 172.21.15.159:0/1468326515 >> 172.21.15.159:0/1468326515 conn(0x7feb840668e0 msgr2=0x7feb84066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 1 -- 172.21.15.159:0/1468326515 shutdown_connections 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.692+0000 7feb8ba63640 1 -- 172.21.15.159:0/1468326515 wait complete. 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 10 monclient: build_initial_monmap 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 10 monclient: monmap: 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:25.698 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 1 librados: starting msgr at 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 1 librados: starting objecter 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 1 Processor -- start 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 1 -- start start 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 1 librados: setting wanted keys 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 1 librados: calling monclient init 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.693+0000 7feb8ba63640 10 monclient: init 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 10 monclient(hunting): picked mon.noname-c con 0x7feb840a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 10 monclient(hunting): picked mon.noname-b con 0x7feb840b4700 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:25.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 10 monclient(hunting): picked mon.noname-a con 0x7feb84142a20 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb84139a80 con 0x7feb84142a20 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8413aaa0 con 0x7feb840b4700 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8413ccb0 con 0x7feb840a96b0 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8ba63640 10 monclient(hunting): authenticate will time out at 5916.004134s 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8a000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7feb84142a20 0x7feb841e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41324/0 (socket says 172.21.15.159:41324) 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.694+0000 7feb8a000640 1 -- 172.21.15.159:0/4172291530 learned_addr learned my addr 172.21.15.159:0/4172291530 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.695+0000 7feb79ffb640 10 client.?.objecter ms_handle_connect 0x7feb84142a20 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.695+0000 7feb79ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.695+0000 7feb79ffb640 10 client.?.objecter ms_handle_connect 0x7feb840b4700 2024-03-25T19:28:25.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.695+0000 7feb79ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.696+0000 7feb79ffb640 10 client.?.objecter ms_handle_connect 0x7feb840a96b0 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.696+0000 7feb79ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140677929 0 0) 0x7feb8413aaa0 con 0x7feb840b4700 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:28:25.701 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb68004eb0 con 0x7feb840b4700 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893773576 0 0) 0x7feb84139a80 con 0x7feb84142a20 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.697+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8413aaa0 con 0x7feb84142a20 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.699+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364225873 0 0) 0x7feb8413ccb0 con 0x7feb840a96b0 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.699+0000 7feb79ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.699+0000 7feb79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.699+0000 7feb79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.699+0000 7feb79ffb640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:28:25.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.699+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb84139a80 con 0x7feb840a96b0 2024-03-25T19:28:25.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146278088 0 0) 0x7feb8413aaa0 con 0x7feb84142a20 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb8413ccb0 con 0x7feb84142a20 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7feb7c001c30 con 0x7feb84142a20 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.704 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1690547337 0 0) 0x7feb68004eb0 con 0x7feb840b4700 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb8413aaa0 con 0x7feb840b4700 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7feb74001b80 con 0x7feb840b4700 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.702+0000 7feb79ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.705 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.706 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.706 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.706 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.706 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.706 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.707 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.704+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013331119 0 0) 0x7feb84139a80 con 0x7feb840a96b0 2024-03-25T19:28:25.707 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.704+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb68004eb0 con 0x7feb840a96b0 2024-03-25T19:28:25.707 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.704+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7feb80001a50 con 0x7feb840a96b0 2024-03-25T19:28:25.707 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.704+0000 7feb79ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.707 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.704+0000 7feb79ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.704+0000 7feb79ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.708 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2162333549 0 0) 0x7feb8413aaa0 con 0x7feb840b4700 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 >> v1:172.21.15.134:6790/0 conn(0x7feb840a96b0 legacy=0x7feb84138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 >> v1:172.21.15.134:6789/0 conn(0x7feb84142a20 legacy=0x7feb841e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 1 monclient: found mon.b 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb84133ac0 con 0x7feb840b4700 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb8ba63640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb8ba63640 10 monclient: _renew_subs 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb8ba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:25.709 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.705+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb841396a0 con 0x7feb840b4700 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.706+0000 7feb8ba63640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.706+0000 7feb8ba63640 10 monclient: _renew_subs 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.706+0000 7feb8ba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.706+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb84136ba0 con 0x7feb840b4700 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.706+0000 7feb8ba63640 1 librados: init done 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.707+0000 7feb8ba63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.707+0000 7feb8ba63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.707+0000 7feb8ba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:25.710 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.707+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb841398e0 con 0x7feb840b4700 2024-03-25T19:28:25.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb74001d40 con 0x7feb840b4700 2024-03-25T19:28:25.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb79ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:25.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7feb74004a90 con 0x7feb840b4700 2024-03-25T19:28:25.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:25.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:25.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb79ffb640 10 monclient: dump: 2024-03-25T19:28:25.746 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:25.747 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:25.747 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.747 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:25.747 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:25.747 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:25.747 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:25.748 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:25.748 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:25.748 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:25.748 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb7b7fe640 4 set_mon_vals no callback set 2024-03-25T19:28:25.748 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.745+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feb74004f80 con 0x7feb840b4700 2024-03-25T19:28:25.752 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:25.752+0000 7f9760632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:28:25.764 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:25.764+0000 7fab1464b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:28:25.810 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:25.810+0000 7fab1464b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:28:25.886 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:25.886+0000 7f9760632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:28:25.924 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:25.924+0000 7fab1464b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3665527024 0 0) 0x7feb740056c0 con 0x7feb840b4700 2024-03-25T19:28:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 10 client.4132.objecter ms_dispatch 0x7feb84000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:28:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:28:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:28:25.926 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T19:28:25.926 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feb74005b50 con 0x7feb840b4700 2024-03-25T19:28:25.926 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:25.926 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:25.925+0000 7feb79ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:26.002 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:26.002+0000 7f9760632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:28:26.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.018+0000 7feb8ba63640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dfd685c1-d3ec-4f5d-9e86-56b1c412f421", "id": 3}] 2024-03-25T19:28:26.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.018+0000 7feb8ba63640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dfd685c1-d3ec-4f5d-9e86-56b1c412f421", "id": 3}] 2024-03-25T19:28:26.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.018+0000 7feb8ba63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:26.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.018+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dfd685c1-d3ec-4f5d-9e86-56b1c412f421", "id": 3} v 0) v1 -- 0x7feb8413edc0 con 0x7feb840b4700 2024-03-25T19:28:26.038 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:26.038+0000 7fab1464b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:28:26.047 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:26.047+0000 7f9760632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:28:26.117 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.117+0000 7feb79ffb640 1 -- 172.21.15.159:0/4172291530 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dfd685c1-d3ec-4f5d-9e86-56b1c412f421", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 654043155 0 525046306) 0x7feb74005230 con 0x7feb840b4700 2024-03-25T19:28:26.117 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.117+0000 7feb79ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dfd685c1-d3ec-4f5d-9e86-56b1c412f421", "id": 3}] 2024-03-25T19:28:26.117 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.117+0000 7feb79ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:26.117 INFO:teuthology.orchestra.run.smithi159.stdout:3 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 10 librados: watch_flush enter 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 10 librados: watch_flush exit 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 10 client.4132.objecter successfully canceled tick 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 10 monclient: shutdown 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 >> v1:172.21.15.159:6789/0 conn(0x7feb840b4700 legacy=0x7feb8413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 shutdown_connections 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 >> 172.21.15.159:0/4172291530 conn(0x7feb840668e0 msgr2=0x7feb840b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 shutdown_connections 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 1 -- 172.21.15.159:0/4172291530 wait complete. 2024-03-25T19:28:26.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.118+0000 7feb8ba63640 1 librados: shutdown 2024-03-25T19:28:26.129 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 4e799e3d-df93-4507-a38b-5e5b552aefb1 4 2024-03-25T19:28:26.166 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:26.167+0000 7f9760632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:26.202 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:26.202+0000 7fab1464b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:26.229 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.229+0000 7fb479d72640 10 monclient: get_monmap_and_config 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.229+0000 7fb479d72640 10 monclient: build_initial_monmap 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.229+0000 7fb479d72640 10 monclient: monmap: 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:26.230724+0000 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:26.230724+0000 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.229+0000 7fb479d72640 1 Processor -- start 2024-03-25T19:28:26.230 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.229+0000 7fb479d72640 1 -- start start 2024-03-25T19:28:26.231 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.229+0000 7fb479d72640 10 monclient: init 2024-03-25T19:28:26.231 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:26.231 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 10 monclient(hunting): picked mon.noname-c con 0x7fb4740b4700 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 10 monclient(hunting): picked mon.noname-b con 0x7fb474142a20 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 10 monclient(hunting): picked mon.noname-a con 0x7fb4740a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4740a9d60 con 0x7fb4740a96b0 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47413ff10 con 0x7fb4740b4700 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4740aa1f0 con 0x7fb474142a20 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.230+0000 7fb479d72640 10 monclient(hunting): authenticate will time out at 5916.540222s 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.231+0000 7fb472ffd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fb474142a20 0x7fb47413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50336/0 (socket says 172.21.15.159:50336) 2024-03-25T19:28:26.232 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.231+0000 7fb472ffd640 1 -- 172.21.15.159:0/451855211 learned_addr learned my addr 172.21.15.159:0/451855211 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445104177 0 0) 0x7fb4740aa1f0 con 0x7fb474142a20 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c003520 con 0x7fb474142a20 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279413568 0 0) 0x7fb47413ff10 con 0x7fb4740b4700 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.234 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4740aa1f0 con 0x7fb4740b4700 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934388492 0 0) 0x7fb4740a9d60 con 0x7fb4740a96b0 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:28:26.235 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.233+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47413ff10 con 0x7fb4740a96b0 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235556340 0 0) 0x7fb45c003520 con 0x7fb474142a20 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 >> v1:172.21.15.159:6789/0 conn(0x7fb4740b4700 legacy=0x7fb474142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 >> v1:172.21.15.134:6789/0 conn(0x7fb4740a96b0 legacy=0x7fb47413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 1 monclient: found mon.noname-b 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4740b4ae0 con 0x7fb474142a20 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:26.237 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb479d72640 5 monclient: authenticate success, global_id 4139 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb479d72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb468001690 con 0x7fb474142a20 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb4727fc640 10 monclient: dump: 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.238 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.239 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.239 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.236+0000 7fb479d72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:26.241 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4680017f0 con 0x7fb474142a20 2024-03-25T19:28:26.241 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb4727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb4727fc640 1 -- 172.21.15.159:0/451855211 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb468001c00 con 0x7fb474142a20 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb4727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb4727fc640 10 monclient: dump: 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb479d72640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:26.242 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.238+0000 7fb479d72640 4 set_mon_vals no callback set 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 10 monclient: shutdown 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 1 -- 172.21.15.159:0/451855211 >> v1:172.21.15.134:6790/0 conn(0x7fb474142a20 legacy=0x7fb47413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 1 -- 172.21.15.159:0/451855211 shutdown_connections 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 1 -- 172.21.15.159:0/451855211 >> 172.21.15.159:0/451855211 conn(0x7fb4740668e0 msgr2=0x7fb474066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 1 -- 172.21.15.159:0/451855211 shutdown_connections 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 1 -- 172.21.15.159:0/451855211 wait complete. 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 10 monclient: build_initial_monmap 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.239+0000 7fb479d72640 10 monclient: monmap: 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 librados: starting msgr at 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 librados: starting objecter 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 Processor -- start 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 -- start start 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 librados: setting wanted keys 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 librados: calling monclient init 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient: init 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient(hunting): picked mon.noname-a con 0x7fb4740a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:26.244 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient(hunting): picked mon.noname-b con 0x7fb4740b4700 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient(hunting): picked mon.noname-c con 0x7fb474142a20 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4740ae500 con 0x7fb4740a96b0 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4740ae7e0 con 0x7fb4740b4700 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4741dbad0 con 0x7fb474142a20 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.240+0000 7fb479d72640 10 monclient(hunting): authenticate will time out at 5916.550634s 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.241+0000 7fb473fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fb474142a20 0x7fb4741e23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50346/0 (socket says 172.21.15.159:50346) 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.241+0000 7fb473fff640 1 -- 172.21.15.159:0/1631073516 learned_addr learned my addr 172.21.15.159:0/1631073516 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.241+0000 7fb45bfff640 10 client.?.objecter ms_handle_connect 0x7fb474142a20 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.241+0000 7fb45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.241+0000 7fb45bfff640 10 client.?.objecter ms_handle_connect 0x7fb4740b4700 2024-03-25T19:28:26.245 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.241+0000 7fb45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.246 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025287668 0 0) 0x7fb4741dbad0 con 0x7fb474142a20 2024-03-25T19:28:26.246 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.246 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.246 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.246 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:28:26.246 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb460004eb0 con 0x7fb474142a20 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178752859 0 0) 0x7fb4740ae7e0 con 0x7fb4740b4700 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.243+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4741dbad0 con 0x7fb4740b4700 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.245+0000 7fb45bfff640 10 client.?.objecter ms_handle_connect 0x7fb4740a96b0 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.245+0000 7fb45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.247+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228968643 0 0) 0x7fb4740ae500 con 0x7fb4740a96b0 2024-03-25T19:28:26.248 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.247+0000 7fb45bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.248 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.247+0000 7fb45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.248 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.247+0000 7fb45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.248 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.247+0000 7fb45bfff640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:28:26.248 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.247+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4740ae7e0 con 0x7fb4740a96b0 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.248+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4250206693 0 0) 0x7fb460004eb0 con 0x7fb474142a20 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.248+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4740ae500 con 0x7fb474142a20 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.248+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb46c001be0 con 0x7fb474142a20 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.248+0000 7fb45bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.248+0000 7fb45bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.248+0000 7fb45bfff640 10 monclient(hunting): dump: 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.249 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.250 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.250 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.250 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.250 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.248+0000 7fb45bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:26.250 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.249+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2391333675 0 0) 0x7fb4741dbad0 con 0x7fb4740b4700 2024-03-25T19:28:26.250 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.249+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb460004eb0 con 0x7fb4740b4700 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.249+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb468001bc0 con 0x7fb4740b4700 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.249+0000 7fb45bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.249+0000 7fb45bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.249+0000 7fb45bfff640 10 monclient(hunting): dump: 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.251 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.252 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:26.250+0000 7fab1464b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:26.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1217959065 0 0) 0x7fb4740ae500 con 0x7fb474142a20 2024-03-25T19:28:26.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 >> v1:172.21.15.159:6789/0 conn(0x7fb4740b4700 legacy=0x7fb474139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 >> v1:172.21.15.134:6789/0 conn(0x7fb4740a96b0 legacy=0x7fb4741db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:26.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 1 monclient: found mon.c 2024-03-25T19:28:26.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:26.253 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47413a560 con 0x7fb474142a20 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 10 monclient: _finish_auth 0 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb45bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 5 monclient: authenticate success, global_id 4142 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 10 monclient: _renew_subs 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47413a7d0 con 0x7fb474142a20 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 10 monclient: _renew_subs 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4741e3ce0 con 0x7fb474142a20 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.252+0000 7fb479d72640 1 librados: init done 2024-03-25T19:28:26.254 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.253+0000 7fb479d72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.255 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.253+0000 7fb479d72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.255 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.253+0000 7fb479d72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.255 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.253+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4741dbc60 con 0x7fb474142a20 2024-03-25T19:28:26.268 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:26.269+0000 7f9760632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:28:26.406 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:26.406+0000 7fab1464b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:28:26.429 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:26.430+0000 7f9760632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:26.444 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb46c0020d0 con 0x7fb474142a20 2024-03-25T19:28:26.444 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb45bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:26.444 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb46c003b30 con 0x7fb474142a20 2024-03-25T19:28:26.444 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb45bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.444 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb45bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:26.444 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb45bfff640 10 monclient: dump: 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb46c0040e0 con 0x7fb474142a20 2024-03-25T19:28:26.445 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.444+0000 7fb471ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.446+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2645339811 0 0) 0x7fb46c0069d0 con 0x7fb474142a20 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.446+0000 7fb45bfff640 10 client.4142.objecter ms_dispatch 0x7fb474000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.446+0000 7fb45bfff640 3 client.4142.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.446+0000 7fb45bfff640 3 client.4142.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.447+0000 7fb45bfff640 20 client.4142.objecter dump_active .. 0 homeless 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.447+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb46c0045c0 con 0x7fb474142a20 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.447+0000 7fb45bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.447+0000 7fb45bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:26.472 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:26.473+0000 7f9760632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:26.520 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.519+0000 7fb479d72640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e799e3d-df93-4507-a38b-5e5b552aefb1", "id": 4}] 2024-03-25T19:28:26.520 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.519+0000 7fb479d72640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e799e3d-df93-4507-a38b-5e5b552aefb1", "id": 4}] 2024-03-25T19:28:26.520 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.519+0000 7fb479d72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.520 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.519+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd new", "uuid": "4e799e3d-df93-4507-a38b-5e5b552aefb1", "id": 4} v 0) v1 -- 0x7fb47404f730 con 0x7fb474142a20 2024-03-25T19:28:26.640 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:26.640+0000 7f9760632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:28:26.653 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.653+0000 7fb45bfff640 1 -- 172.21.15.159:0/1631073516 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e799e3d-df93-4507-a38b-5e5b552aefb1", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2011930328 0 1697012583) 0x7fb46c0064c0 con 0x7fb474142a20 2024-03-25T19:28:26.653 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.653+0000 7fb45bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e799e3d-df93-4507-a38b-5e5b552aefb1", "id": 4}] 2024-03-25T19:28:26.653 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.653+0000 7fb45bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:26.654 INFO:teuthology.orchestra.run.smithi159.stdout:4 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 10 librados: watch_flush enter 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 10 librados: watch_flush exit 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 20 client.4142.objecter shutdown clearing up homeless session... 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 10 client.4142.objecter successfully canceled tick 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 10 monclient: shutdown 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 >> v1:172.21.15.134:6790/0 conn(0x7fb474142a20 legacy=0x7fb4741e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.655 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 shutdown_connections 2024-03-25T19:28:26.656 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 >> 172.21.15.159:0/1631073516 conn(0x7fb4740668e0 msgr2=0x7fb47413c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:26.656 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 shutdown_connections 2024-03-25T19:28:26.656 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 1 -- 172.21.15.159:0/1631073516 wait complete. 2024-03-25T19:28:26.656 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.654+0000 7fb479d72640 1 librados: shutdown 2024-03-25T19:28:26.665 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 538f3a54-921e-4264-8bfd-11837dc6ce63 5 2024-03-25T19:28:26.764 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.764+0000 7f3ec46c1640 10 monclient: get_monmap_and_config 2024-03-25T19:28:26.764 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.764+0000 7f3ec46c1640 10 monclient: build_initial_monmap 2024-03-25T19:28:26.764 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.764+0000 7f3ec46c1640 10 monclient: monmap: 2024-03-25T19:28:26.764 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:26.764 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.764 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:26.766020+0000 2024-03-25T19:28:26.764 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:26.766020+0000 2024-03-25T19:28:26.765 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:26.765 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.765 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:26.765 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:26.765 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:26.765 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.766 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 1 Processor -- start 2024-03-25T19:28:26.766 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 1 -- start start 2024-03-25T19:28:26.766 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient: init 2024-03-25T19:28:26.766 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:26.766 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient(hunting): picked mon.noname-b con 0x7f3ebc0b4700 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient(hunting): picked mon.noname-c con 0x7f3ebc142a20 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient(hunting): picked mon.noname-a con 0x7f3ebc0a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc0a9d60 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc13ff10 con 0x7f3ebc142a20 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc0aa1f0 con 0x7f3ebc0b4700 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.765+0000 7f3ec46c1640 10 monclient(hunting): authenticate will time out at 5917.075609s 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.766+0000 7f3ec1c5c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3ebc142a20 0x7f3ebc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:47374/0 (socket says 172.21.15.159:47374) 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.766+0000 7f3ec245d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f3ebc0b4700 0x7f3ebc142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50360/0 (socket says 172.21.15.159:50360) 2024-03-25T19:28:26.767 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.766+0000 7f3ec1c5c640 1 -- 172.21.15.159:0/1831629713 learned_addr learned my addr 172.21.15.159:0/1831629713 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:26.768 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.768+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214048681 0 0) 0x7f3ebc0aa1f0 con 0x7f3ebc0b4700 2024-03-25T19:28:26.768 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.768+0000 7f3ec145b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.768 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.768+0000 7f3ec145b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.768 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.768+0000 7f3ec145b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.768 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.768+0000 7f3ec145b640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:28:26.768 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.768+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea8003520 con 0x7f3ebc0b4700 2024-03-25T19:28:26.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171609043 0 0) 0x7f3ebc0a9d60 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:28:26.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc0aa1f0 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.769 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420388773 0 0) 0x7f3ebc13ff10 con 0x7f3ebc142a20 2024-03-25T19:28:26.770 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.770 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.770 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.770 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:28:26.770 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.769+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc0a9d60 con 0x7f3ebc142a20 2024-03-25T19:28:26.772 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520267165 0 0) 0x7f3ea8003520 con 0x7f3ebc0b4700 2024-03-25T19:28:26.772 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 >> v1:172.21.15.159:6789/0 conn(0x7f3ebc142a20 legacy=0x7f3ebc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.772 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 >> v1:172.21.15.134:6789/0 conn(0x7f3ebc0a96b0 legacy=0x7f3ebc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.772 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 10 monclient: _finish_hunting 0 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 1 monclient: found mon.noname-b 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc0b4ae0 con 0x7f3ebc0b4700 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 10 monclient: _finish_auth 0 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 10 monclient: _check_auth_tickets 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3eb8001690 con 0x7f3ebc0b4700 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec46c1640 5 monclient: authenticate success, global_id 4145 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec46c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec145b640 10 monclient: dump: 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.773 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.772+0000 7f3ec46c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:26.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3eb80017f0 con 0x7f3ebc0b4700 2024-03-25T19:28:26.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec145b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:26.777 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec145b640 1 -- 172.21.15.159:0/1831629713 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3eb8001c00 con 0x7f3ebc0b4700 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec145b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec145b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec145b640 10 monclient: dump: 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.778 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 4 set_mon_vals no callback set 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 10 monclient: shutdown 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 1 -- 172.21.15.159:0/1831629713 >> v1:172.21.15.134:6790/0 conn(0x7f3ebc0b4700 legacy=0x7f3ebc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 1 -- 172.21.15.159:0/1831629713 shutdown_connections 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 1 -- 172.21.15.159:0/1831629713 >> 172.21.15.159:0/1831629713 conn(0x7f3ebc0668e0 msgr2=0x7f3ebc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 1 -- 172.21.15.159:0/1831629713 shutdown_connections 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.774+0000 7f3ec46c1640 1 -- 172.21.15.159:0/1831629713 wait complete. 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 10 monclient: build_initial_monmap 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 10 monclient: monmap: 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 1 librados: starting msgr at 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 1 librados: starting objecter 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 1 Processor -- start 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 1 -- start start 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 1 librados: setting wanted keys 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 1 librados: calling monclient init 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.775+0000 7f3ec46c1640 10 monclient: init 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 10 monclient(hunting): picked mon.noname-c con 0x7f3ebc0a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:26.780 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 10 monclient(hunting): picked mon.noname-b con 0x7f3ebc0b4700 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 10 monclient(hunting): picked mon.noname-a con 0x7f3ebc142a20 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc0adb40 con 0x7f3ebc142a20 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc131970 con 0x7f3ebc0b4700 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc0af460 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec46c1640 10 monclient(hunting): authenticate will time out at 5917.086204s 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec245d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f3ebc0a96b0 0x7f3ebc0afb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:50370/0 (socket says 172.21.15.159:50370) 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.776+0000 7f3ec245d640 1 -- 172.21.15.159:0/2360688859 learned_addr learned my addr 172.21.15.159:0/2360688859 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.777+0000 7f3eb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3ebc0a96b0 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.777+0000 7f3eb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.777+0000 7f3eb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3ebc0b4700 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.777+0000 7f3eb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.777+0000 7f3eb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3ebc142a20 2024-03-25T19:28:26.781 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.777+0000 7f3eb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.782 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.778+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334820239 0 0) 0x7f3ebc0af460 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.778+0000 7f3eb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.778+0000 7f3eb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.778+0000 7f3eb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.778+0000 7f3eb2ffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.778+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea0004eb0 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350469118 0 0) 0x7f3ebc0adb40 con 0x7f3ebc142a20 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc0af460 con 0x7f3ebc142a20 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780927251 0 0) 0x7f3ebc131970 con 0x7f3ebc0b4700 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.783 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.784 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.784 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:28:26.784 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.779+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc0adb40 con 0x7f3ebc0b4700 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.783+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 773869124 0 0) 0x7f3ea0004eb0 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.783+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ebc131970 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.783+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3eb8001bc0 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.783+0000 7f3eb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.783+0000 7f3eb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.783+0000 7f3eb2ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.783+0000 7f3eb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3226098953 0 0) 0x7f3ebc0af460 con 0x7f3ebc142a20 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ea0004eb0 con 0x7f3ebc142a20 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3eb4001b50 con 0x7f3ebc142a20 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.786 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1826645568 0 0) 0x7f3ebc0adb40 con 0x7f3ebc0b4700 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ebc0af460 con 0x7f3ebc0b4700 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3eac001830 con 0x7f3ebc0b4700 2024-03-25T19:28:26.787 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.784+0000 7f3eb2ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.788 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3726570890 0 0) 0x7f3ebc131970 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 >> v1:172.21.15.159:6789/0 conn(0x7f3ebc0b4700 legacy=0x7f3ebc130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 >> v1:172.21.15.134:6789/0 conn(0x7f3ebc142a20 legacy=0x7f3ebc131260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 1 monclient: found mon.c 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc0af740 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:26.790 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3eb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 5 monclient: authenticate success, global_id 4148 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 10 monclient: _renew_subs 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ebc0addb0 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 10 monclient: _renew_subs 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ebc0ad610 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.787+0000 7f3ec46c1640 1 librados: init done 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.788+0000 7f3ec46c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.788+0000 7f3ec46c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.788+0000 7f3ec46c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.791 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.788+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ebc0adff0 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.792 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3eb80011f0 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.792 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:26.792 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3eb8004b10 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.792 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.792 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:26.792 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 10 monclient: dump: 2024-03-25T19:28:26.792 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3eb8005080 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3ec0c5a640 4 set_mon_vals no callback set 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 643735914 0 0) 0x7f3eb8006820 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 10 client.4148.objecter ms_dispatch 0x7f3ebc000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:28:26.793 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 3 client.4148.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:28:26.794 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 3 client.4148.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:28:26.794 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.790+0000 7f3eb2ffd640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-25T19:28:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.887+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3eb8006350 con 0x7f3ebc0a96b0 2024-03-25T19:28:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.887+0000 7f3eb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.887+0000 7f3eb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:26.958 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.958+0000 7f3ec46c1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "538f3a54-921e-4264-8bfd-11837dc6ce63", "id": 5}] 2024-03-25T19:28:26.958 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.958+0000 7f3ec46c1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "538f3a54-921e-4264-8bfd-11837dc6ce63", "id": 5}] 2024-03-25T19:28:26.958 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.958+0000 7f3ec46c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:26.958 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:26.958+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd new", "uuid": "538f3a54-921e-4264-8bfd-11837dc6ce63", "id": 5} v 0) v1 -- 0x7f3ebc0ad910 con 0x7f3ebc0a96b0 2024-03-25T19:28:27.021 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.020+0000 7f3eb2ffd640 1 -- 172.21.15.159:0/2360688859 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "538f3a54-921e-4264-8bfd-11837dc6ce63", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2380658928 0 1988412176) 0x7f3eb8006670 con 0x7f3ebc0a96b0 2024-03-25T19:28:27.021 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.020+0000 7f3eb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "538f3a54-921e-4264-8bfd-11837dc6ce63", "id": 5}] 2024-03-25T19:28:27.021 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.020+0000 7f3eb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:27.021 INFO:teuthology.orchestra.run.smithi159.stdout:5 2024-03-25T19:28:27.022 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.021+0000 7f3ec46c1640 10 librados: watch_flush enter 2024-03-25T19:28:27.022 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.021+0000 7f3ec46c1640 10 librados: watch_flush exit 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.021+0000 7f3ec46c1640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.021+0000 7f3ec46c1640 10 client.4148.objecter successfully canceled tick 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 10 monclient: shutdown 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 >> v1:172.21.15.134:6790/0 conn(0x7f3ebc0a96b0 legacy=0x7f3ebc0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 shutdown_connections 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 >> 172.21.15.159:0/2360688859 conn(0x7f3ebc0668e0 msgr2=0x7f3ebc0b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 shutdown_connections 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 1 -- 172.21.15.159:0/2360688859 wait complete. 2024-03-25T19:28:27.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.022+0000 7f3ec46c1640 1 librados: shutdown 2024-03-25T19:28:27.033 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 93761c0a-2dcf-45f2-acae-3d602d251675 6 2024-03-25T19:28:27.132 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.132+0000 7f7121cb3640 10 monclient: get_monmap_and_config 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.132+0000 7f7121cb3640 10 monclient: build_initial_monmap 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.132+0000 7f7121cb3640 10 monclient: monmap: 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:27.134241+0000 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:27.134241+0000 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:27.133 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.133+0000 7f7121cb3640 1 Processor -- start 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.133+0000 7f7121cb3640 1 -- start start 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.133+0000 7f7121cb3640 10 monclient: init 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f711c0b4700 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f711c142a20 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f711c0a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711c0a9d60 con 0x7f711c0b4700 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711c13ff10 con 0x7f711c142a20 2024-03-25T19:28:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711c0aa1f0 con 0x7f711c0a96b0 2024-03-25T19:28:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7121cb3640 10 monclient(hunting): authenticate will time out at 5917.444037s 2024-03-25T19:28:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7113fff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f711c142a20 0x7f711c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:47390/0 (socket says 172.21.15.159:47390) 2024-03-25T19:28:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.134+0000 7f7113fff640 1 -- 172.21.15.159:0/3182818931 learned_addr learned my addr 172.21.15.159:0/3182818931 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507976078 0 0) 0x7f711c0aa1f0 con 0x7f711c0a96b0 2024-03-25T19:28:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:28:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7108003520 con 0x7f711c0a96b0 2024-03-25T19:28:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498592360 0 0) 0x7f711c0a9d60 con 0x7f711c0b4700 2024-03-25T19:28:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:28:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711c0aa1f0 con 0x7f711c0b4700 2024-03-25T19:28:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820494108 0 0) 0x7f711c13ff10 con 0x7f711c142a20 2024-03-25T19:28:27.139 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.139 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.139 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.139 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:28:27.139 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.137+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711c0a9d60 con 0x7f711c142a20 2024-03-25T19:28:27.140 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539619556 0 0) 0x7f711c0aa1f0 con 0x7f711c0b4700 2024-03-25T19:28:27.140 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 >> v1:172.21.15.134:6790/0 conn(0x7f711c0a96b0 legacy=0x7f711c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.140 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 >> v1:172.21.15.159:6789/0 conn(0x7f711c142a20 legacy=0x7f711c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.140 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 1 monclient: found mon.noname-a 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711c0b4ae0 con 0x7f711c0b4700 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f711affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f7121cb3640 5 monclient: authenticate success, global_id 4152 2024-03-25T19:28:27.141 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.140+0000 7f7121cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:27.142 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f710c001630 con 0x7f711c0b4700 2024-03-25T19:28:27.142 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.142 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:27.142 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 10 monclient: dump: 2024-03-25T19:28:27.142 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f710c001880 con 0x7f711c0b4700 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f7121cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f7121cb3640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 1 -- 172.21.15.159:0/3182818931 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f710c001c10 con 0x7f711c0b4700 2024-03-25T19:28:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f7121cb3640 4 set_mon_vals no callback set 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.142+0000 7f711affd640 10 monclient: dump: 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.144 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.147 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.143+0000 7f7121cb3640 10 monclient: shutdown 2024-03-25T19:28:27.147 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.143+0000 7f7121cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:27.147 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.143+0000 7f7121cb3640 1 -- 172.21.15.159:0/3182818931 >> v1:172.21.15.134:6789/0 conn(0x7f711c0b4700 legacy=0x7f711c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.147 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.143+0000 7f7121cb3640 1 -- 172.21.15.159:0/3182818931 shutdown_connections 2024-03-25T19:28:27.147 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.143+0000 7f7121cb3640 1 -- 172.21.15.159:0/3182818931 >> 172.21.15.159:0/3182818931 conn(0x7f711c0668e0 msgr2=0x7f711c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:27.147 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.143+0000 7f7121cb3640 1 -- 172.21.15.159:0/3182818931 shutdown_connections 2024-03-25T19:28:27.147 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.143+0000 7f7121cb3640 1 -- 172.21.15.159:0/3182818931 wait complete. 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 10 monclient: build_initial_monmap 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 10 monclient: monmap: 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 1 librados: starting msgr at 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 1 librados: starting objecter 2024-03-25T19:28:27.148 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 1 Processor -- start 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 1 -- start start 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 1 librados: setting wanted keys 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 1 librados: calling monclient init 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.144+0000 7f7121cb3640 10 monclient: init 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f711c0a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f711c0b4700 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f711c142a20 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711c13e830 con 0x7f711c0a96b0 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711c1292e0 con 0x7f711c142a20 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711c1295c0 con 0x7f711c0b4700 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.145+0000 7f7121cb3640 10 monclient(hunting): authenticate will time out at 5917.455335s 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f711bfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f711c142a20 0x7f711c12a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:47392/0 (socket says 172.21.15.159:47392) 2024-03-25T19:28:27.149 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f711bfff640 1 -- 172.21.15.159:0/1245304863 learned_addr learned my addr 172.21.15.159:0/1245304863 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:27.150 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f711b7fe640 1 --1- 172.21.15.159:0/1245304863 >> v1:172.21.15.134:6789/0 conn(0x7f711c0a96b0 0x7f711c132000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:41368/0 (socket says 172.21.15.159:41368) 2024-03-25T19:28:27.150 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f7118ff9640 10 client.?.objecter ms_handle_connect 0x7f711c0a96b0 2024-03-25T19:28:27.150 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f7118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:27.150 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f7118ff9640 10 client.?.objecter ms_handle_connect 0x7f711c142a20 2024-03-25T19:28:27.150 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f7118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:27.150 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f7118ff9640 10 client.?.objecter ms_handle_connect 0x7f711c0b4700 2024-03-25T19:28:27.150 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.146+0000 7f7118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:27.151 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288455947 0 0) 0x7f711c13e830 con 0x7f711c0a96b0 2024-03-25T19:28:27.151 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.151 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.151 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.151 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:28:27.151 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70fc004eb0 con 0x7f711c0a96b0 2024-03-25T19:28:27.151 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955477823 0 0) 0x7f711c1295c0 con 0x7f711c0b4700 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711c13e830 con 0x7f711c0b4700 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410348726 0 0) 0x7f711c1292e0 con 0x7f711c142a20 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:28:27.152 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.148+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711c1295c0 con 0x7f711c142a20 2024-03-25T19:28:27.153 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3090368869 0 0) 0x7f711c13e830 con 0x7f711c0b4700 2024-03-25T19:28:27.153 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f711c1292e0 con 0x7f711c0b4700 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f71040018f0 con 0x7f711c0b4700 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.154 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743102097 0 0) 0x7f70fc004eb0 con 0x7f711c0a96b0 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f711c13e830 con 0x7f711c0a96b0 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f710c001630 con 0x7f711c0a96b0 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.152+0000 7f7118ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.155 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.153+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3775665371 0 0) 0x7f711c1295c0 con 0x7f711c142a20 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.153+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70fc004eb0 con 0x7f711c142a20 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.153+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7114001c30 con 0x7f711c142a20 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.153+0000 7f7118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.153+0000 7f7118ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.153+0000 7f7118ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.156 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.157 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.157 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.157 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.157 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.158 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2226943816 0 0) 0x7f711c1292e0 con 0x7f711c0b4700 2024-03-25T19:28:27.158 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 >> v1:172.21.15.159:6789/0 conn(0x7f711c142a20 legacy=0x7f711c12a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.158 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 >> v1:172.21.15.134:6789/0 conn(0x7f711c0a96b0 legacy=0x7f711c132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.158 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:27.158 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 1 monclient: found mon.c 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711c1298a0 con 0x7f711c0b4700 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7118ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 5 monclient: authenticate success, global_id 4154 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 10 monclient: _renew_subs 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f711c129b80 con 0x7f711c0b4700 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 10 monclient: _renew_subs 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f711c12adc0 con 0x7f711c0b4700 2024-03-25T19:28:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.156+0000 7f7121cb3640 1 librados: init done 2024-03-25T19:28:27.160 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.157+0000 7f7121cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:27.160 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.157+0000 7f7121cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:27.160 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.157+0000 7f7121cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:27.160 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.157+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711c127f50 con 0x7f711c0b4700 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7104001d40 con 0x7f711c0b4700 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f71040049c0 con 0x7f711c0b4700 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 10 monclient: dump: 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.161 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7104004f70 con 0x7f711c0b4700 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f711a7fc640 4 set_mon_vals no callback set 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2232785799 0 0) 0x7f7104006780 con 0x7f711c0b4700 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 10 client.4154.objecter ms_dispatch 0x7f711c000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 3 client.4154.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 3 client.4154.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:28:27.162 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.160+0000 7f7118ff9640 20 client.4154.objecter dump_active .. 0 homeless 2024-03-25T19:28:27.250 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.250+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f71040062a0 con 0x7f711c0b4700 2024-03-25T19:28:27.251 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.250+0000 7f7118ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:27.251 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.250+0000 7f7118ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:27.279 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:27.279+0000 7fab1464b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:28:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.322+0000 7f7121cb3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "93761c0a-2dcf-45f2-acae-3d602d251675", "id": 6}] 2024-03-25T19:28:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.322+0000 7f7121cb3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "93761c0a-2dcf-45f2-acae-3d602d251675", "id": 6}] 2024-03-25T19:28:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.322+0000 7f7121cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.322+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd new", "uuid": "93761c0a-2dcf-45f2-acae-3d602d251675", "id": 6} v 0) v1 -- 0x7f711c04f730 con 0x7f711c0b4700 2024-03-25T19:28:27.382 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.381+0000 7f7118ff9640 1 -- 172.21.15.159:0/1245304863 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93761c0a-2dcf-45f2-acae-3d602d251675", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 901776119 0 1113811849) 0x7f7104006530 con 0x7f711c0b4700 2024-03-25T19:28:27.382 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.381+0000 7f7118ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "93761c0a-2dcf-45f2-acae-3d602d251675", "id": 6}] 2024-03-25T19:28:27.382 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.381+0000 7f7118ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:27.382 INFO:teuthology.orchestra.run.smithi159.stdout:6 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 10 librados: watch_flush enter 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 10 librados: watch_flush exit 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 20 client.4154.objecter shutdown clearing up homeless session... 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 10 client.4154.objecter successfully canceled tick 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 10 monclient: shutdown 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 >> v1:172.21.15.134:6790/0 conn(0x7f711c0b4700 legacy=0x7f711c129fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 shutdown_connections 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 >> 172.21.15.159:0/1245304863 conn(0x7f711c0668e0 msgr2=0x7f711c0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:27.384 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 shutdown_connections 2024-03-25T19:28:27.385 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 1 -- 172.21.15.159:0/1245304863 wait complete. 2024-03-25T19:28:27.385 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.383+0000 7f7121cb3640 1 librados: shutdown 2024-03-25T19:28:27.394 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 519ac8d9-2636-4167-97a5-3feca8638ff8 7 2024-03-25T19:28:27.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.492+0000 7fbe98c2a640 10 monclient: get_monmap_and_config 2024-03-25T19:28:27.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.492+0000 7fbe98c2a640 10 monclient: build_initial_monmap 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.492+0000 7fbe98c2a640 10 monclient: monmap: 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:28:27.494101+0000 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:28:27.494101+0000 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:27.493 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 1 Processor -- start 2024-03-25T19:28:27.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 1 -- start start 2024-03-25T19:28:27.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 10 monclient: init 2024-03-25T19:28:27.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:27.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe940b4700 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe94142a20 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe940a96b0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.493+0000 7fbe98c2a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940a9d60 con 0x7fbe940b4700 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.494+0000 7fbe98c2a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9413ff10 con 0x7fbe940a96b0 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.494+0000 7fbe98c2a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940aa1f0 con 0x7fbe94142a20 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.494+0000 7fbe98c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.494+0000 7fbe98c2a640 10 monclient(hunting): authenticate will time out at 5917.803724s 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.494+0000 7fbe92d9d640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbe940a96b0 0x7fbe9413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:47404/0 (socket says 172.21.15.159:47404) 2024-03-25T19:28:27.496 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.494+0000 7fbe92d9d640 1 -- 172.21.15.159:0/286103958 learned_addr learned my addr 172.21.15.159:0/286103958 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:27.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.496+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588116911 0 0) 0x7fbe9413ff10 con 0x7fbe940a96b0 2024-03-25T19:28:27.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.496+0000 7fbe9159a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.496+0000 7fbe9159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.496+0000 7fbe9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.496+0000 7fbe9159a640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:28:27.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.496+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe80003520 con 0x7fbe940a96b0 2024-03-25T19:28:27.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.498+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284910578 0 0) 0x7fbe940aa1f0 con 0x7fbe94142a20 2024-03-25T19:28:27.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.498+0000 7fbe9159a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.498+0000 7fbe9159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.498+0000 7fbe9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.498+0000 7fbe9159a640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:28:27.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.498+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9413ff10 con 0x7fbe94142a20 2024-03-25T19:28:27.500 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2253637101 0 0) 0x7fbe80003520 con 0x7fbe940a96b0 2024-03-25T19:28:27.500 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 >> v1:172.21.15.134:6790/0 conn(0x7fbe94142a20 legacy=0x7fbe9413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.500 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 >> v1:172.21.15.134:6789/0 conn(0x7fbe940b4700 legacy=0x7fbe94142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.500 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 10 monclient: _finish_hunting 0 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 1 monclient: found mon.noname-c 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940b4ae0 con 0x7fbe940a96b0 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 10 monclient: _finish_auth 0 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 10 monclient: _check_auth_tickets 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe8c001a10 con 0x7fbe940a96b0 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe9159a640 10 monclient: dump: 2024-03-25T19:28:27.501 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe98c2a640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.500+0000 7fbe98c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe8c001b70 con 0x7fbe940a96b0 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe9159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe9159a640 1 -- 172.21.15.159:0/286103958 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe8c001f80 con 0x7fbe940a96b0 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe9159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe9159a640 10 monclient: dump: 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe98c2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:27.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.509+0000 7fbe98c2a640 4 set_mon_vals no callback set 2024-03-25T19:28:27.513 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.510+0000 7fbe98c2a640 10 monclient: shutdown 2024-03-25T19:28:27.513 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.510+0000 7fbe98c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:27.513 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.510+0000 7fbe98c2a640 1 -- 172.21.15.159:0/286103958 >> v1:172.21.15.159:6789/0 conn(0x7fbe940a96b0 legacy=0x7fbe9413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.510+0000 7fbe98c2a640 1 -- 172.21.15.159:0/286103958 shutdown_connections 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.510+0000 7fbe98c2a640 1 -- 172.21.15.159:0/286103958 >> 172.21.15.159:0/286103958 conn(0x7fbe940668e0 msgr2=0x7fbe94066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.510+0000 7fbe98c2a640 1 -- 172.21.15.159:0/286103958 shutdown_connections 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.510+0000 7fbe98c2a640 1 -- 172.21.15.159:0/286103958 wait complete. 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 10 monclient: build_initial_monmap 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 10 monclient: monmap: 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:27.514 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 1 librados: starting msgr at 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 1 librados: starting objecter 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 1 Processor -- start 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 1 -- start start 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 1 librados: setting wanted keys 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 1 librados: calling monclient init 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.511+0000 7fbe98c2a640 10 monclient: init 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:27.515 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe940a96b0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe940b4700 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe94142a20 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9413b3e0 con 0x7fbe940b4700 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940ae520 con 0x7fbe940a96b0 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940ab880 con 0x7fbe94142a20 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe98c2a640 10 monclient(hunting): authenticate will time out at 5917.821812s 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe9259c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbe940a96b0 0x7fbe940ac810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:47406/0 (socket says 172.21.15.159:47406) 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe9259c640 1 -- 172.21.15.159:0/3886539607 learned_addr learned my addr 172.21.15.159:0/3886539607 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe8affd640 10 client.?.objecter ms_handle_connect 0x7fbe94142a20 2024-03-25T19:28:27.516 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.512+0000 7fbe8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:27.517 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.513+0000 7fbe8affd640 10 client.?.objecter ms_handle_connect 0x7fbe940a96b0 2024-03-25T19:28:27.517 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.513+0000 7fbe8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:27.524 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.524+0000 7fbe8affd640 10 client.?.objecter ms_handle_connect 0x7fbe940b4700 2024-03-25T19:28:27.524 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.524+0000 7fbe8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:27.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.555+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490505238 0 0) 0x7fbe940ae520 con 0x7fbe940a96b0 2024-03-25T19:28:27.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.555+0000 7fbe8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.555+0000 7fbe8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.555+0000 7fbe8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.555 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.555+0000 7fbe8affd640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:28:27.556 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.555+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe74004eb0 con 0x7fbe940a96b0 2024-03-25T19:28:27.560 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.560+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3852805273 0 0) 0x7fbe74004eb0 con 0x7fbe940a96b0 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.560+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe940ae520 con 0x7fbe940a96b0 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.560+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe84001a20 con 0x7fbe940a96b0 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.560+0000 7fbe8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.560+0000 7fbe8affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.560+0000 7fbe8affd640 10 monclient(hunting): dump: 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.560+0000 7fbe8affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.561+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063406967 0 0) 0x7fbe9413b3e0 con 0x7fbe940b4700 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.561+0000 7fbe8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.561+0000 7fbe8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.561+0000 7fbe8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.561+0000 7fbe8affd640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:28:27.562 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.561+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe74004eb0 con 0x7fbe940b4700 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.563+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075734657 0 0) 0x7fbe940ab880 con 0x7fbe94142a20 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.563+0000 7fbe8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.563+0000 7fbe8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.563+0000 7fbe8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.563+0000 7fbe8affd640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.563+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9413b3e0 con 0x7fbe94142a20 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1468832269 0 0) 0x7fbe940ae520 con 0x7fbe940a96b0 2024-03-25T19:28:27.565 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 >> v1:172.21.15.134:6790/0 conn(0x7fbe94142a20 legacy=0x7fbe940b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 >> v1:172.21.15.134:6789/0 conn(0x7fbe940b4700 legacy=0x7fbe94139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 1 monclient: found mon.b 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940aba10 con 0x7fbe940a96b0 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 10 monclient: _renew_subs 2024-03-25T19:28:27.566 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9413a490 con 0x7fbe940a96b0 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 10 monclient: _renew_subs 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe940ad430 con 0x7fbe940a96b0 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.564+0000 7fbe98c2a640 1 librados: init done 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.565+0000 7fbe98c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.565+0000 7fbe98c2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.565+0000 7fbe98c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.565+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9404f730 con 0x7fbe940a96b0 2024-03-25T19:28:27.568 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:27.565+0000 7fab1464b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:28:27.618 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:27.618+0000 7fab1464b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe84001ca0 con 0x7fbe940a96b0 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe84003a60 con 0x7fbe940a96b0 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 10 monclient: dump: 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T19:28:27.664 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbe84004010 con 0x7fbe940a96b0 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe90d99640 4 set_mon_vals no callback set 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1570963288 0 0) 0x7fbe840068a0 con 0x7fbe940a96b0 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 10 client.4156.objecter ms_dispatch 0x7fbe94000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbe84018e00 con 0x7fbe940a96b0 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:27.665 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.663+0000 7fbe8affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:27.688 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:27.688+0000 7f9760632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:28:27.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.736+0000 7fbe98c2a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "519ac8d9-2636-4167-97a5-3feca8638ff8", "id": 7}] 2024-03-25T19:28:27.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.736+0000 7fbe98c2a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "519ac8d9-2636-4167-97a5-3feca8638ff8", "id": 7}] 2024-03-25T19:28:27.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.736+0000 7fbe98c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:27.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.736+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd new", "uuid": "519ac8d9-2636-4167-97a5-3feca8638ff8", "id": 7} v 0) v1 -- 0x7fbe94142220 con 0x7fbe940a96b0 2024-03-25T19:28:27.738 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:27.738+0000 7fab1464b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:28:27.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.849+0000 7fbe8affd640 1 -- 172.21.15.159:0/3886539607 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "519ac8d9-2636-4167-97a5-3feca8638ff8", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3470127753 0 1371665406) 0x7fbe84006220 con 0x7fbe940a96b0 2024-03-25T19:28:27.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.849+0000 7fbe8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "519ac8d9-2636-4167-97a5-3feca8638ff8", "id": 7}] 2024-03-25T19:28:27.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.849+0000 7fbe8affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:27.849 INFO:teuthology.orchestra.run.smithi159.stdout:7 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 10 librados: watch_flush enter 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 10 librados: watch_flush exit 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 10 monclient: shutdown 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 >> v1:172.21.15.159:6789/0 conn(0x7fbe940a96b0 legacy=0x7fbe940ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 shutdown_connections 2024-03-25T19:28:27.852 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 >> 172.21.15.159:0/3886539607 conn(0x7fbe940668e0 msgr2=0x7fbe9413c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:27.852 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 shutdown_connections 2024-03-25T19:28:27.852 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 1 -- 172.21.15.159:0/3886539607 wait complete. 2024-03-25T19:28:27.852 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T19:28:27.850+0000 7fbe98c2a640 1 librados: shutdown 2024-03-25T19:28:27.860 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.861 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:28:27.861 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:28:27.863 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:28:27.863 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.863 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:28:27.863 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:28:27.865 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:28:27.865 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.865 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:28:27.866 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:28:27.867 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:28:27.868 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.868 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:28:27.868 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:28:27.870 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:28:27.870 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.870 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:28:27.871 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:28:27.873 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:28:27.873 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.873 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:28:27.873 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:28:27.875 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:28:27.875 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.875 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:28:27.875 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:28:27.877 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:28:27.877 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:28:27.877 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:28:27.878 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:28:27.879 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:28:27.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.043+0000 7fbb6119f640 10 monclient: get_monmap_and_config 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.043+0000 7fbb6119f640 10 monclient: build_initial_monmap 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.043+0000 7fbb6119f640 10 monclient: monmap: 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:28:28.044152+0000 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:28:28.044152+0000 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:28.044 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:28.044 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:28.044 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:28.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.051+0000 7fbb6119f640 1 Processor -- start 2024-03-25T19:28:28.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.051+0000 7fbb6119f640 1 -- start start 2024-03-25T19:28:28.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.051+0000 7fbb6119f640 10 monclient: init 2024-03-25T19:28:28.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 10 monclient(hunting): picked mon.noname-b con 0x7fbb5c0b1f20 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 10 monclient(hunting): picked mon.noname-a con 0x7fbb5c146a90 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 10 monclient(hunting): picked mon.noname-c con 0x7fbb5c146ef0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c071e70 con 0x7fbb5c146a90 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c147340 con 0x7fbb5c146ef0 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c06f320 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.052+0000 7fbb6119f640 10 monclient(hunting): authenticate will time out at 5940.211006s 2024-03-25T19:28:28.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.053+0000 7fbb6099e640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbb5c146ef0 0x7fbb5c152520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52762/0 (socket says 172.21.15.134:52762) 2024-03-25T19:28:28.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.053+0000 7fbb6099e640 1 -- 172.21.15.134:0/770847186 learned_addr learned my addr 172.21.15.134:0/770847186 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:28.070 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:28.071+0000 7f9760632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:28:28.162 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:28.162+0000 7f9760632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:28:28.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.176+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083350221 0 0) 0x7fbb5c071e70 con 0x7fbb5c146a90 2024-03-25T19:28:28.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.176+0000 7fbb5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:28.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.176+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:28.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.176+0000 7fbb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:28.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.176+0000 7fbb5affd640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:28:28.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.176+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb44003520 con 0x7fbb5c146a90 2024-03-25T19:28:28.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3341811276 0 0) 0x7fbb44003520 con 0x7fbb5c146a90 2024-03-25T19:28:28.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 >> v1:172.21.15.134:6790/0 conn(0x7fbb5c0b1f20 legacy=0x7fbb5c140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:28.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 >> v1:172.21.15.159:6789/0 conn(0x7fbb5c146ef0 legacy=0x7fbb5c152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:28.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:28.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 1 monclient: found mon.noname-a 2024-03-25T19:28:28.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:28.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c144030 con 0x7fbb5c146a90 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb6119f640 5 monclient: authenticate success, global_id 4164 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb6119f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbb50001690 con 0x7fbb5c146a90 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb5affd640 10 monclient: dump: 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:28.189 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:28.190 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.188+0000 7fbb6119f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:28.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.193+0000 7fbb5affd640 1 -- 172.21.15.134:0/770847186 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb500018b0 con 0x7fbb5c146a90 2024-03-25T19:28:28.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.193+0000 7fbb5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:28.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.193+0000 7fbb6119f640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:28.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.193+0000 7fbb6119f640 4 set_mon_vals no callback set 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.194+0000 7fbb6119f640 10 monclient: shutdown 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.194+0000 7fbb6119f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.194+0000 7fbb6119f640 1 -- 172.21.15.134:0/770847186 >> v1:172.21.15.134:6789/0 conn(0x7fbb5c146a90 legacy=0x7fbb5c14edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.194+0000 7fbb6119f640 1 -- 172.21.15.134:0/770847186 shutdown_connections 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.194+0000 7fbb6119f640 1 -- 172.21.15.134:0/770847186 >> 172.21.15.134:0/770847186 conn(0x7fbb5c066ac0 msgr2=0x7fbb5c066ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.194+0000 7fbb6119f640 1 -- 172.21.15.134:0/770847186 shutdown_connections 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.194+0000 7fbb6119f640 1 -- 172.21.15.134:0/770847186 wait complete. 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.195+0000 7fbb6119f640 10 monclient: build_initial_monmap 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.195+0000 7fbb6119f640 10 monclient: monmap: 2024-03-25T19:28:28.196 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.196+0000 7fbb6119f640 1 librados: starting msgr at 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.196+0000 7fbb6119f640 1 librados: starting objecter 2024-03-25T19:28:28.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.196+0000 7fbb6119f640 1 Processor -- start 2024-03-25T19:28:28.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.196+0000 7fbb6119f640 1 -- start start 2024-03-25T19:28:28.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.196+0000 7fbb6119f640 1 librados: setting wanted keys 2024-03-25T19:28:28.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.196+0000 7fbb6119f640 1 librados: calling monclient init 2024-03-25T19:28:28.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.196+0000 7fbb6119f640 10 monclient: init 2024-03-25T19:28:28.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:28.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:28.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 10 monclient(hunting): picked mon.noname-b con 0x7fbb5c0b1f20 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:28.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 10 monclient(hunting): picked mon.noname-c con 0x7fbb5c146a90 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:28.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 10 monclient(hunting): picked mon.noname-a con 0x7fbb5c146ef0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:28.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c1409b0 con 0x7fbb5c146ef0 2024-03-25T19:28:28.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c0aba90 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c0aafa0 con 0x7fbb5c146a90 2024-03-25T19:28:28.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:28.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.197+0000 7fbb6119f640 10 monclient(hunting): authenticate will time out at 5940.356388s 2024-03-25T19:28:28.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.207+0000 7fbb5bfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbb5c0b1f20 0x7fbb5c0aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52770/0 (socket says 172.21.15.134:52770) 2024-03-25T19:28:28.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.207+0000 7fbb5bfff640 1 -- 172.21.15.134:0/1474526903 learned_addr learned my addr 172.21.15.134:0/1474526903 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:28.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.213+0000 7fbb58ff9640 10 client.?.objecter ms_handle_connect 0x7fbb5c146ef0 2024-03-25T19:28:28.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.213+0000 7fbb58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:28.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.216+0000 7fbb58ff9640 10 client.?.objecter ms_handle_connect 0x7fbb5c0b1f20 2024-03-25T19:28:28.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.216+0000 7fbb58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:28.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.218+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320650090 0 0) 0x7fbb5c1409b0 con 0x7fbb5c146ef0 2024-03-25T19:28:28.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.218+0000 7fbb58ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:28.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.218+0000 7fbb58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:28.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.219+0000 7fbb58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:28.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.219+0000 7fbb58ff9640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:28:28.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.219+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb48004eb0 con 0x7fbb5c146ef0 2024-03-25T19:28:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.224+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173278306 0 0) 0x7fbb5c0aba90 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.224+0000 7fbb58ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.224+0000 7fbb58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.224+0000 7fbb58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.224+0000 7fbb58ff9640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:28:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.224+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c1409b0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.231+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1899651457 0 0) 0x7fbb48004eb0 con 0x7fbb5c146ef0 2024-03-25T19:28:28.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.231+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb5c0aba90 con 0x7fbb5c146ef0 2024-03-25T19:28:28.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.233+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1965785654 0 0) 0x7fbb5c1409b0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.233+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb48004eb0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.233+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbb4c001950 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.233+0000 7fbb58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.233+0000 7fbb58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.233+0000 7fbb58ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:28.233 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:28.234 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:28.234 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:28.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.233+0000 7fbb58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:28.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2225353582 0 0) 0x7fbb48004eb0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 >> v1:172.21.15.134:6790/0 conn(0x7fbb5c146a90 legacy=0x7fbb5c1ddce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:28.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 >> v1:172.21.15.134:6789/0 conn(0x7fbb5c146ef0 legacy=0x7fbb5c1e46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 1 monclient: found mon.b 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c0abce0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 5 monclient: authenticate success, global_id 4162 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 10 monclient: _renew_subs 2024-03-25T19:28:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:28.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5c0abfc0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:28.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 10 monclient: _renew_subs 2024-03-25T19:28:28.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:28.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5c0aabd0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.238+0000 7fbb6119f640 1 librados: init done 2024-03-25T19:28:28.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.240+0000 7fbb6119f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:28.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.240+0000 7fbb6119f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:28.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.240+0000 7fbb6119f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:28.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.240+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5c0627b0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.360 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T19:28:28.360+0000 7f9760632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:28:28.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.430+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb4c001ab0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.430+0000 7fbb58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:28.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.430+0000 7fbb5a7fc640 4 set_mon_vals no callback set 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.431+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbb4c0038c0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.431+0000 7fbb58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.431+0000 7fbb58ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.431+0000 7fbb58ff9640 10 monclient: dump: 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:28:28.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.431+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3763768414 0 0) 0x7fbb4c003ae0 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fbb4c015780 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 10 client.4162.objecter ms_dispatch 0x7fbb5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 3 client.4162.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 3 client.4162.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbb4c003d90 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:28.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.434+0000 7fbb58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:28.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.548+0000 7fbb6119f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:28.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.548+0000 7fbb6119f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:28.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.548+0000 7fbb6119f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:28.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.548+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb5c070270 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.653+0000 7fbb58ff9640 1 -- 172.21.15.134:0/1474526903 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fbb4c013220 con 0x7fbb5c0b1f20 2024-03-25T19:28:28.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.653+0000 7fbb58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:28.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.653+0000 7fbb58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:28.653 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T19:28:28.653 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 10 librados: watch_flush enter 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 10 librados: watch_flush exit 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 10 client.4162.objecter successfully canceled tick 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 10 monclient: shutdown 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 >> v1:172.21.15.159:6789/0 conn(0x7fbb5c0b1f20 legacy=0x7fbb5c0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 shutdown_connections 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.654+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 >> 172.21.15.134:0/1474526903 conn(0x7fbb5c066ac0 msgr2=0x7fbb5c145f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.655+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 shutdown_connections 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.655+0000 7fbb6119f640 1 -- 172.21.15.134:0/1474526903 wait complete. 2024-03-25T19:28:28.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T19:28:28.655+0000 7fbb6119f640 1 librados: shutdown 2024-03-25T19:28:28.670 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:28:28.671 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:28:29.477 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:29.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.577+0000 7f27554a8640 10 monclient: get_monmap_and_config 2024-03-25T19:28:29.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.577+0000 7f27554a8640 10 monclient: build_initial_monmap 2024-03-25T19:28:29.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.577+0000 7f27554a8640 10 monclient: monmap: 2024-03-25T19:28:29.577 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:28:29.578134+0000 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:28:29.578134+0000 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.577+0000 7f27554a8640 1 Processor -- start 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.577+0000 7f27554a8640 1 -- start start 2024-03-25T19:28:29.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.577+0000 7f27554a8640 10 monclient: init 2024-03-25T19:28:29.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 10 monclient(hunting): picked mon.noname-a con 0x7f27500b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 10 monclient(hunting): picked mon.noname-b con 0x7f27500b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 10 monclient(hunting): picked mon.noname-c con 0x7f275014be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500af940 con 0x7f27500b4d60 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500b0a70 con 0x7f275014be90 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500b0d50 con 0x7f27500b1730 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.578+0000 7f27554a8640 10 monclient(hunting): authenticate will time out at 5941.737068s 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.579+0000 7f274effd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f27500b4d60 0x7f275013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:38452/0 (socket says 172.21.15.134:38452) 2024-03-25T19:28:29.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.579+0000 7f274effd640 1 -- 172.21.15.134:0/3242227001 learned_addr learned my addr 172.21.15.134:0/3242227001 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:29.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809630012 0 0) 0x7f27500b0d50 con 0x7f27500b1730 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): my global_id is 4172 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2738003520 con 0x7f27500b1730 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451887227 0 0) 0x7f27500af940 con 0x7f27500b4d60 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:29.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 10 monclient(hunting): my global_id is 4188 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.582+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27500b0d50 con 0x7f27500b4d60 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.583+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99555128 0 0) 0x7f27500b0a70 con 0x7f275014be90 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.583+0000 7f274dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.583+0000 7f274dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.583+0000 7f274dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.583+0000 7f274dffb640 10 monclient(hunting): my global_id is 4174 2024-03-25T19:28:29.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.583+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27500af940 con 0x7f275014be90 2024-03-25T19:28:29.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679320602 0 0) 0x7f2738003520 con 0x7f27500b1730 2024-03-25T19:28:29.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 >> v1:172.21.15.159:6789/0 conn(0x7f275014be90 legacy=0x7f275014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:29.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 >> v1:172.21.15.134:6789/0 conn(0x7f27500b4d60 legacy=0x7f275013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:29.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:29.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 1 monclient: found mon.noname-b 2024-03-25T19:28:29.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:29.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500b1030 con 0x7f27500b1730 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2744001690 con 0x7f27500b1730 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f27554a8640 5 monclient: authenticate success, global_id 4172 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f27554a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f274dffb640 10 monclient: dump: 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:29.587 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.586+0000 7f27554a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27440017f0 con 0x7f27500b1730 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f274dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f274dffb640 1 -- 172.21.15.134:0/3242227001 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2744001c00 con 0x7f27500b1730 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f27554a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f27554a8640 4 set_mon_vals no callback set 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f274dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f274dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.588+0000 7f274dffb640 10 monclient: dump: 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:29.590 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.589+0000 7f27554a8640 10 monclient: shutdown 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.589+0000 7f27554a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.589+0000 7f27554a8640 1 -- 172.21.15.134:0/3242227001 >> v1:172.21.15.134:6790/0 conn(0x7f27500b1730 legacy=0x7f27500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.589+0000 7f27554a8640 1 -- 172.21.15.134:0/3242227001 shutdown_connections 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.589+0000 7f27554a8640 1 -- 172.21.15.134:0/3242227001 >> 172.21.15.134:0/3242227001 conn(0x7f27500617a0 msgr2=0x7f2750061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.590+0000 7f27554a8640 1 -- 172.21.15.134:0/3242227001 shutdown_connections 2024-03-25T19:28:29.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.590+0000 7f27554a8640 1 -- 172.21.15.134:0/3242227001 wait complete. 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.590+0000 7f27554a8640 10 monclient: build_initial_monmap 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.590+0000 7f27554a8640 10 monclient: monmap: 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:29.594 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.591+0000 7f27554a8640 1 librados: starting msgr at 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.591+0000 7f27554a8640 1 librados: starting objecter 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.591+0000 7f27554a8640 1 Processor -- start 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.591+0000 7f27554a8640 1 -- start start 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.591+0000 7f27554a8640 1 librados: setting wanted keys 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.591+0000 7f27554a8640 1 librados: calling monclient init 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.591+0000 7f27554a8640 10 monclient: init 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 10 monclient(hunting): picked mon.noname-c con 0x7f27500b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:29.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 10 monclient(hunting): picked mon.noname-b con 0x7f27500b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 10 monclient(hunting): picked mon.noname-a con 0x7f275014be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2750138250 con 0x7f275014be90 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27501383e0 con 0x7f27500b4d60 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27501387a0 con 0x7f27500b1730 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f27554a8640 10 monclient(hunting): authenticate will time out at 5941.750914s 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f274effd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f27500b1730 0x7f2750137b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:49536/0 (socket says 172.21.15.134:49536) 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.592+0000 7f274effd640 1 -- 172.21.15.134:0/3072628331 learned_addr learned my addr 172.21.15.134:0/3072628331 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.593+0000 7f27377fe640 10 client.?.objecter ms_handle_connect 0x7f27500b1730 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.593+0000 7f27377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.593+0000 7f27377fe640 10 client.?.objecter ms_handle_connect 0x7f275014be90 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.593+0000 7f27377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.593+0000 7f27377fe640 10 client.?.objecter ms_handle_connect 0x7f27500b4d60 2024-03-25T19:28:29.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.593+0000 7f27377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.595+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560307857 0 0) 0x7f27501387a0 con 0x7f27500b1730 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.595+0000 7f27377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.595+0000 7f27377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.595+0000 7f27377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.595+0000 7f27377fe640 10 monclient(hunting): my global_id is 4175 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.595+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f273c004eb0 con 0x7f27500b1730 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063719774 0 0) 0x7f27501383e0 con 0x7f27500b4d60 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:29.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27501387a0 con 0x7f27500b4d60 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832318259 0 0) 0x7f2750138250 con 0x7f275014be90 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 10 monclient(hunting): my global_id is 4191 2024-03-25T19:28:29.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.596+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27501383e0 con 0x7f275014be90 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.601+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953551510 0 0) 0x7f273c004eb0 con 0x7f27500b1730 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.601+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2750138250 con 0x7f27500b1730 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.601+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f27400019f0 con 0x7f27500b1730 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.601+0000 7f27377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.601+0000 7f27377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.601+0000 7f27377fe640 10 monclient(hunting): dump: 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.601 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:29.602 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:29.602 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:29.602 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:29.602 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:29.602 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.601+0000 7f27377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:29.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1908140780 0 0) 0x7f27501383e0 con 0x7f275014be90 2024-03-25T19:28:29.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f273c004eb0 con 0x7f275014be90 2024-03-25T19:28:29.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2748001bd0 con 0x7f275014be90 2024-03-25T19:28:29.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient(hunting): dump: 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 101319090 0 0) 0x7f27501387a0 con 0x7f27500b4d60 2024-03-25T19:28:29.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27501383e0 con 0x7f27500b4d60 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2744001b00 con 0x7f27500b4d60 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient(hunting): dump: 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2435362638 0 0) 0x7f2750138250 con 0x7f27500b1730 2024-03-25T19:28:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 >> v1:172.21.15.159:6789/0 conn(0x7f27500b4d60 legacy=0x7f27501db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 >> v1:172.21.15.134:6789/0 conn(0x7f275014be90 legacy=0x7f2750130e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 1 monclient: found mon.c 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.604+0000 7f27377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750138930 con 0x7f27500b1730 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27377fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27377fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 5 monclient: authenticate success, global_id 4175 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 10 monclient: _renew_subs 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2750143b60 con 0x7f27500b1730 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 10 monclient: _renew_subs 2024-03-25T19:28:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:29.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27501dbb50 con 0x7f27500b1730 2024-03-25T19:28:29.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.605+0000 7f27554a8640 1 librados: init done 2024-03-25T19:28:29.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.606+0000 7f27554a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:29.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.606+0000 7f27554a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:29.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.606+0000 7f27554a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:29.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.606+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27500627b0 con 0x7f27500b1730 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.808+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27400047e0 con 0x7f27500b1730 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.808+0000 7f27377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.808+0000 7f274d7fa640 4 set_mon_vals no callback set 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.809+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2740004af0 con 0x7f27500b1730 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.809+0000 7f27377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.809+0000 7f27377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.809+0000 7f27377fe640 10 monclient: dump: 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.810 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:29.811 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:29.811 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:29.811 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:29.811 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:29.811 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:29.811 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:29.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.809+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 3) v1 ==== 89499+0+0 (unknown 1149214149 0 0) 0x7f274001aa40 con 0x7f27500b1730 2024-03-25T19:28:29.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.814+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f27400547e0 con 0x7f27500b1730 2024-03-25T19:28:29.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.814+0000 7f27377fe640 10 client.4175.objecter ms_dispatch 0x7f2750000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:29.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.814+0000 7f27377fe640 3 client.4175.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:29.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.814+0000 7f27377fe640 3 client.4175.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:29.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.814+0000 7f27377fe640 20 client.4175.objecter dump_active .. 0 homeless 2024-03-25T19:28:29.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.814+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2740050120 con 0x7f27500b1730 2024-03-25T19:28:29.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.814+0000 7f27377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:29.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.815+0000 7f27377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:29.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.939+0000 7f27554a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:29.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.939+0000 7f27554a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:29.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.939+0000 7f27554a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:29.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:29.939+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2750069f40 con 0x7f27500b1730 2024-03-25T19:28:30.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.046+0000 7f27377fe640 1 -- 172.21.15.134:0/3072628331 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f2740052220 con 0x7f27500b1730 2024-03-25T19:28:30.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.046+0000 7f27377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:30.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.046+0000 7f27377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:30.047 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:28:30.047 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:30.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 10 librados: watch_flush enter 2024-03-25T19:28:30.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 10 librados: watch_flush exit 2024-03-25T19:28:30.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 20 client.4175.objecter shutdown clearing up homeless session... 2024-03-25T19:28:30.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 10 client.4175.objecter successfully canceled tick 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 10 monclient: shutdown 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 >> v1:172.21.15.134:6790/0 conn(0x7f27500b1730 legacy=0x7f2750137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 shutdown_connections 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 >> 172.21.15.134:0/3072628331 conn(0x7f27500617a0 msgr2=0x7f27500b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 shutdown_connections 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 1 -- 172.21.15.134:0/3072628331 wait complete. 2024-03-25T19:28:30.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:30.047+0000 7f27554a8640 1 librados: shutdown 2024-03-25T19:28:30.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:30.824 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:30.824+0000 7faab5237640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:28:30.884 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T19:28:30.884+0000 7faab5237640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:28:34.747 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:06.808 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:34.807 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:06.868 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:34.810 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:06.873 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:34.845 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:06.903 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:34.870 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:06.933 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:34.931 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:06.989 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:34.964 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:07.023 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.023 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:07.081 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.228 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:07.277 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.291 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:07.340 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.329 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:07.371 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.377 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-25T19:28:35.371+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:35.392 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:07.433 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.462 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-25T19:28:35.454+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:35.495 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-25T19:28:35.487+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:35.562 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:07.611 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.576 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-25T19:28:35.570+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:35.592 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:07.648 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.620 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:07.670 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.653 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:07.710 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.830 INFO:tasks.ceph.osd.2.smithi134.stderr:2024-03-25T19:28:35.824+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:35.838 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:07.900 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.928 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:07.991 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:35.977 INFO:tasks.ceph.osd.1.smithi134.stderr:2024-03-25T19:28:35.969+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:35.996 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:08.054 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.055 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:08.113 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.183 INFO:tasks.ceph.osd.0.smithi134.stderr:2024-03-25T19:28:36.172+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:36.223 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:08.284 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.232 INFO:tasks.ceph.osd.3.smithi134.stderr:2024-03-25T19:28:36.225+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:28:36.302 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:08.365 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.304 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:08.353 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.365 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:08.423 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.422 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:08.480 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.465 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:08.506 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.537 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:08.599 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.656 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:08.719 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.670 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:08.719 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.687 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:08.743 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.687 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:08.736 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.732 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:08.790 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.742 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:08.801 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.802 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:08.864 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.827 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:08.869 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.868 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:36.944 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.007 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.955+0000 7f53fce4a640 10 monclient: get_monmap_and_config 2024-03-25T19:28:36.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.955+0000 7f53fce4a640 10 monclient: build_initial_monmap 2024-03-25T19:28:36.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.955+0000 7f53fce4a640 10 monclient: monmap: 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:28:36.956973+0000 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:28:36.956973+0000 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:36.956 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.956+0000 7f53fce4a640 1 Processor -- start 2024-03-25T19:28:36.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.956+0000 7f53fce4a640 1 -- start start 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.956+0000 7f53fce4a640 10 monclient: init 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 10 monclient(hunting): picked mon.noname-c con 0x7f53f80ae6e0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 10 monclient(hunting): picked mon.noname-b con 0x7f53f806f080 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 10 monclient(hunting): picked mon.noname-a con 0x7f53f80704e0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f80718d0 con 0x7f53f80704e0 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f806fa30 con 0x7f53f80ae6e0 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f8070090 con 0x7f53f806f080 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53fce4a640 10 monclient(hunting): authenticate will time out at 5949.115970s 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53f659c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f53f80ae6e0 0x7f53f813c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:48124/0 (socket says 172.21.15.134:48124) 2024-03-25T19:28:36.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.957+0000 7f53f659c640 1 -- 172.21.15.134:0/2976502898 learned_addr learned my addr 172.21.15.134:0/2976502898 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707799118 0 0) 0x7f53f8070090 con 0x7f53f806f080 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e0005730 con 0x7f53f806f080 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087630621 0 0) 0x7f53f806fa30 con 0x7f53f80ae6e0 2024-03-25T19:28:36.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f8070090 con 0x7f53f80ae6e0 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148413779 0 0) 0x7f53f80718d0 con 0x7f53f80704e0 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:28:36.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.960+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f806fa30 con 0x7f53f80704e0 2024-03-25T19:28:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351740397 0 0) 0x7f53f8070090 con 0x7f53f80ae6e0 2024-03-25T19:28:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 >> v1:172.21.15.134:6790/0 conn(0x7f53f806f080 legacy=0x7f53f8071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 >> v1:172.21.15.134:6789/0 conn(0x7f53f80704e0 legacy=0x7f53f814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 10 monclient: _finish_hunting 0 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 1 monclient: found mon.noname-c 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f806f510 con 0x7f53f80ae6e0 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 10 monclient: _finish_auth 0 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 10 monclient: _check_auth_tickets 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f53e8001670 con 0x7f53f80ae6e0 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53f559a640 10 monclient: dump: 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.964 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53fce4a640 5 monclient: authenticate success, global_id 4204 2024-03-25T19:28:36.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.963+0000 7f53fce4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.964+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53e80017d0 con 0x7f53f80ae6e0 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.964+0000 7f53f559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.964+0000 7f53fce4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.964+0000 7f53fce4a640 4 set_mon_vals no callback set 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53f559a640 1 -- 172.21.15.134:0/2976502898 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f53e8001c20 con 0x7f53f80ae6e0 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53f559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53f559a640 10 monclient: dump: 2024-03-25T19:28:36.968 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53fce4a640 10 monclient: shutdown 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53fce4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53fce4a640 1 -- 172.21.15.134:0/2976502898 >> v1:172.21.15.159:6789/0 conn(0x7f53f80ae6e0 legacy=0x7f53f813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:36.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53fce4a640 1 -- 172.21.15.134:0/2976502898 shutdown_connections 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53fce4a640 1 -- 172.21.15.134:0/2976502898 >> 172.21.15.134:0/2976502898 conn(0x7f53f806b630 msgr2=0x7f53f80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53fce4a640 1 -- 172.21.15.134:0/2976502898 shutdown_connections 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.965+0000 7f53fce4a640 1 -- 172.21.15.134:0/2976502898 wait complete. 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 10 monclient: build_initial_monmap 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 10 monclient: monmap: 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:36.970 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 1 librados: starting msgr at 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 1 librados: starting objecter 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 1 Processor -- start 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 1 -- start start 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 1 librados: setting wanted keys 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 1 librados: calling monclient init 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.966+0000 7f53fce4a640 10 monclient: init 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 10 monclient(hunting): picked mon.noname-c con 0x7f53f80704e0 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 10 monclient(hunting): picked mon.noname-a con 0x7f53f80ae6e0 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 10 monclient(hunting): picked mon.noname-b con 0x7f53f813e810 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f8139db0 con 0x7f53f80ae6e0 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f813cdf0 con 0x7f53f813e810 2024-03-25T19:28:36.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f813fd80 con 0x7f53f80704e0 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53fce4a640 10 monclient(hunting): authenticate will time out at 5949.126116s 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53f5d9b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f53f80ae6e0 0x7f53f813e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:52032/0 (socket says 172.21.15.134:52032) 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.967+0000 7f53f5d9b640 1 -- 172.21.15.134:0/269009153 learned_addr learned my addr 172.21.15.134:0/269009153 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.968+0000 7f53e6ffd640 10 client.?.objecter ms_handle_connect 0x7f53f80ae6e0 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.968+0000 7f53e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.968+0000 7f53e6ffd640 10 client.?.objecter ms_handle_connect 0x7f53f813e810 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.968+0000 7f53e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.968+0000 7f53e6ffd640 10 client.?.objecter ms_handle_connect 0x7f53f80704e0 2024-03-25T19:28:36.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.968+0000 7f53e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:36.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240424602 0 0) 0x7f53f813cdf0 con 0x7f53f813e810 2024-03-25T19:28:36.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:36.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:36.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:36.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:28:36.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d8004eb0 con 0x7f53f813e810 2024-03-25T19:28:36.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366518640 0 0) 0x7f53f8139db0 con 0x7f53f80ae6e0 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f813cdf0 con 0x7f53f80ae6e0 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161221427 0 0) 0x7f53f813fd80 con 0x7f53f80704e0 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:28:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.970+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f8139db0 con 0x7f53f80704e0 2024-03-25T19:28:36.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.974+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2759205549 0 0) 0x7f53d8004eb0 con 0x7f53f813e810 2024-03-25T19:28:36.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.974+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f813fd80 con 0x7f53f813e810 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.974+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f53f0001cb0 con 0x7f53f813e810 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.974+0000 7f53e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.974+0000 7f53e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.974+0000 7f53e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.974+0000 7f53e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 674147014 0 0) 0x7f53f8139db0 con 0x7f53f80704e0 2024-03-25T19:28:36.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53d8004eb0 con 0x7f53f80704e0 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f53e8001ae0 con 0x7f53f80704e0 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3673146511 0 0) 0x7f53f813cdf0 con 0x7f53f80ae6e0 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f8139db0 con 0x7f53f80ae6e0 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f53ec001a70 con 0x7f53f80ae6e0 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.975+0000 7f53e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:36.978 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:36.979 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2215192561 0 0) 0x7f53f813fd80 con 0x7f53f813e810 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 >> v1:172.21.15.134:6790/0 conn(0x7f53f80704e0 legacy=0x7f53f8131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 >> v1:172.21.15.134:6789/0 conn(0x7f53f80ae6e0 legacy=0x7f53f813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 1 monclient: found mon.b 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f81324b0 con 0x7f53f813e810 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 10 monclient: _renew_subs 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f81d39f0 con 0x7f53f813e810 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 10 monclient: _renew_subs 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f81df830 con 0x7f53f813e810 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.977+0000 7f53fce4a640 1 librados: init done 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.978+0000 7f53fce4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.978+0000 7f53fce4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.978+0000 7f53fce4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.978+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f81d3130 con 0x7f53f813e810 2024-03-25T19:28:36.988 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:09.037 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:36.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.992+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53f0003990 con 0x7f53f813e810 2024-03-25T19:28:36.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.992+0000 7f53e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:36.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.992+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f53f0003ca0 con 0x7f53f813e810 2024-03-25T19:28:36.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.992+0000 7f53e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:36.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.992+0000 7f53e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:36.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.992+0000 7f53e6ffd640 10 monclient: dump: 2024-03-25T19:28:36.992 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:36.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.992+0000 7f53f4d99640 4 set_mon_vals no callback set 2024-03-25T19:28:36.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.993+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f53f0019c70 con 0x7f53f813e810 2024-03-25T19:28:36.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.993+0000 7f53f659c640 10 monclient: get_auth_request con 0x7f53d806e080 auth_method 0 2024-03-25T19:28:36.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.994+0000 7f53e6ffd640 10 client.4207.objecter ms_handle_connect 0x7f53d806e080 2024-03-25T19:28:36.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.995+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f53f0053970 con 0x7f53f813e810 2024-03-25T19:28:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.995+0000 7f53e6ffd640 10 client.4207.objecter ms_dispatch 0x7f53f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.995+0000 7f53e6ffd640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.995+0000 7f53e6ffd640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:36.995+0000 7f53e6ffd640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T19:28:37.009 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.067 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.030 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.088 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.064 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:09.113 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.066 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.123 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.070 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:09.132 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.157 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:09.199 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.233+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53f004f480 con 0x7f53f813e810 2024-03-25T19:28:37.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.233+0000 7f53e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:37.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.233+0000 7f53e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:37.240 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.303 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.264 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:09.313 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.275 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.333 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.332 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.390 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.348 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:09.410 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.385 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:09.435 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.390 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.447 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.410+0000 7f53fce4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:37.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.410+0000 7f53fce4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:37.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.410+0000 7f53fce4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:37.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.410+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53f80b1c10 con 0x7f53f813e810 2024-03-25T19:28:37.422 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:09.464 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.428+0000 7f53e6ffd640 1 -- 172.21.15.134:0/269009153 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f53f00513b0 con 0x7f53f813e810 2024-03-25T19:28:37.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.428+0000 7f53e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:37.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.428+0000 7f53e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:37.428 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:28:37.428 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 10 librados: watch_flush enter 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 10 librados: watch_flush exit 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 10 client.4207.objecter successfully canceled tick 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 >> v1:172.21.15.159:6800/1875874887 conn(0x7f53d806e080 legacy=0x7f53d8070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 10 monclient: shutdown 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 >> v1:172.21.15.159:6789/0 conn(0x7f53f813e810 legacy=0x7f53f81d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 shutdown_connections 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 >> 172.21.15.134:0/269009153 conn(0x7f53f806b630 msgr2=0x7f53f8071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 shutdown_connections 2024-03-25T19:28:37.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 1 -- 172.21.15.134:0/269009153 wait complete. 2024-03-25T19:28:37.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:37.430+0000 7f53fce4a640 1 librados: shutdown 2024-03-25T19:28:37.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:37.528 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:09.577 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.539 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.602 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.542 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.600 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.613 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:09.675 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.629 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.687 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.657 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:09.706 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.665 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.721 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.690 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:09.732 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.792 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:09.841 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.813 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.871 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.835 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.898 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.881 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:09.943 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.928 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:09.977 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.929 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.987 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.937 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.991 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:37.961 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:10.003 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.056 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:10.105 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.081 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:10.139 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.122 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:10.185 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.153 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:10.215 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.196 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:10.246 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.217 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:10.275 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.220 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:10.276 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.228 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:10.270 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.324 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:10.373 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.349 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:10.407 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.411 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:10.474 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.464 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:10.514 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.498 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:10.554 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.506 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:10.548 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.507 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:10.565 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.589 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:10.638 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.623 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:10.684 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.677 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:10.738 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.733 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:10.782 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.766 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:10.823 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.776 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:10.818 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.824 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:10.882 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.878 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:10.937 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:38.968 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:11.031 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.000 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:11.049 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.023 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:11.086 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.051 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:11.107 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.063 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:11.112 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.064 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:11.122 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.119 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:11.177 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.131 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:11.181 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.274 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:11.316 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.329 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:11.370 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.500 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:11.549 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.591 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:11.640 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.594 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:11.651 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:39.649 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:11.705 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.196 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:16.245 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.247 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.393+0000 7fee6eb9e640 10 monclient: get_monmap_and_config 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.393+0000 7fee6eb9e640 10 monclient: build_initial_monmap 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.393+0000 7fee6eb9e640 10 monclient: monmap: 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:28:44.394948+0000 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:28:44.394948+0000 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:44.394 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:44.395 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.394+0000 7fee6eb9e640 1 Processor -- start 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.395+0000 7fee6eb9e640 1 -- start start 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.395+0000 7fee6eb9e640 10 monclient: init 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 10 monclient(hunting): picked mon.noname-c con 0x7fee680b48e0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 10 monclient(hunting): picked mon.noname-a con 0x7fee6813fc60 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:44.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 10 monclient(hunting): picked mon.noname-b con 0x7fee680b3f90 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:44.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680a9d10 con 0x7fee6813fc60 2024-03-25T19:28:44.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680ab000 con 0x7fee680b48e0 2024-03-25T19:28:44.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680a9680 con 0x7fee680b3f90 2024-03-25T19:28:44.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:44.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6eb9e640 10 monclient(hunting): authenticate will time out at 5956.554992s 2024-03-25T19:28:44.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6d39b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fee6813fc60 0x7fee68145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39448/0 (socket says 172.21.15.134:39448) 2024-03-25T19:28:44.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6d39b640 1 -- 172.21.15.134:0/4073593465 learned_addr learned my addr 172.21.15.134:0/4073593465 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:44.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.396+0000 7fee6db9c640 1 --1- 172.21.15.134:0/4073593465 >> v1:172.21.15.159:6789/0 conn(0x7fee680b48e0 0x7fee6813f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54786/0 (socket says 172.21.15.134:54786) 2024-03-25T19:28:44.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.400+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975886854 0 0) 0x7fee680a9d10 con 0x7fee6813fc60 2024-03-25T19:28:44.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.400+0000 7fee6cb9a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:44.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.400+0000 7fee6cb9a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.400+0000 7fee6cb9a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.400+0000 7fee6cb9a640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.401+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee54003520 con 0x7fee6813fc60 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.401+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530148346 0 0) 0x7fee680ab000 con 0x7fee680b48e0 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.401+0000 7fee6cb9a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.401+0000 7fee6cb9a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.401+0000 7fee6cb9a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.401+0000 7fee6cb9a640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:28:44.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.401+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee680a9d10 con 0x7fee680b48e0 2024-03-25T19:28:44.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.402+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961932166 0 0) 0x7fee680a9680 con 0x7fee680b3f90 2024-03-25T19:28:44.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.402+0000 7fee6cb9a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:44.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.402+0000 7fee6cb9a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:44.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.402+0000 7fee6cb9a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:44.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.402+0000 7fee6cb9a640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:28:44.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.402+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee680ab000 con 0x7fee680b3f90 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4261004025 0 0) 0x7fee54003520 con 0x7fee6813fc60 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 >> v1:172.21.15.134:6790/0 conn(0x7fee680b3f90 legacy=0x7fee6814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 >> v1:172.21.15.159:6789/0 conn(0x7fee680b48e0 legacy=0x7fee6813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 10 monclient: _finish_hunting 0 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 1 monclient: found mon.noname-a 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:28:44.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee680a9810 con 0x7fee6813fc60 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 10 monclient: _finish_auth 0 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 10 monclient: _check_auth_tickets 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee58001690 con 0x7fee6813fc60 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6cb9a640 10 monclient: dump: 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:44.410 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:44.411 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:44.411 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:44.411 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:44.411 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6eb9e640 5 monclient: authenticate success, global_id 4224 2024-03-25T19:28:44.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.409+0000 7fee6eb9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:44.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.411+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee580017f0 con 0x7fee6813fc60 2024-03-25T19:28:44.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.411+0000 7fee6cb9a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:44.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.411+0000 7fee6eb9e640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:44.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.411+0000 7fee6eb9e640 4 set_mon_vals no callback set 2024-03-25T19:28:44.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.412+0000 7fee6cb9a640 1 -- 172.21.15.134:0/4073593465 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee58001c40 con 0x7fee6813fc60 2024-03-25T19:28:44.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.412+0000 7fee6cb9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:44.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.412+0000 7fee6cb9a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:44.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.412+0000 7fee6cb9a640 10 monclient: dump: 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.412+0000 7fee6eb9e640 10 monclient: shutdown 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.412+0000 7fee6eb9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.412+0000 7fee6eb9e640 1 -- 172.21.15.134:0/4073593465 >> v1:172.21.15.134:6789/0 conn(0x7fee6813fc60 legacy=0x7fee68145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:44.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.413+0000 7fee6eb9e640 1 -- 172.21.15.134:0/4073593465 shutdown_connections 2024-03-25T19:28:44.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.413+0000 7fee6eb9e640 1 -- 172.21.15.134:0/4073593465 >> 172.21.15.134:0/4073593465 conn(0x7fee680615c0 msgr2=0x7fee680619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:44.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.413+0000 7fee6eb9e640 1 -- 172.21.15.134:0/4073593465 shutdown_connections 2024-03-25T19:28:44.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.414+0000 7fee6eb9e640 1 -- 172.21.15.134:0/4073593465 wait complete. 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 10 monclient: build_initial_monmap 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 10 monclient: monmap: 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:44.417 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 1 librados: starting msgr at 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 1 librados: starting objecter 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 1 Processor -- start 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 1 -- start start 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 1 librados: setting wanted keys 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 1 librados: calling monclient init 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.415+0000 7fee6eb9e640 10 monclient: init 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 10 monclient(hunting): picked mon.noname-c con 0x7fee680b3f90 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 10 monclient(hunting): picked mon.noname-b con 0x7fee680b48e0 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:44.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 10 monclient(hunting): picked mon.noname-a con 0x7fee6813fc60 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee681db350 con 0x7fee6813fc60 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680af5e0 con 0x7fee680b48e0 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.416+0000 7fee6eb9e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee681dae30 con 0x7fee680b3f90 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee6eb9e640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee6eb9e640 10 monclient(hunting): authenticate will time out at 5956.575667s 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee6d39b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fee680b48e0 0x7fee681d6ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54800/0 (socket says 172.21.15.134:54800) 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee6d39b640 1 -- 172.21.15.134:0/122092160 learned_addr learned my addr 172.21.15.134:0/122092160 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee5e7fc640 10 client.?.objecter ms_handle_connect 0x7fee680b48e0 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee5e7fc640 10 client.?.objecter ms_handle_connect 0x7fee6813fc60 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee5e7fc640 10 client.?.objecter ms_handle_connect 0x7fee680b3f90 2024-03-25T19:28:44.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.417+0000 7fee5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233393840 0 0) 0x7fee681dae30 con 0x7fee680b3f90 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee4c004eb0 con 0x7fee680b3f90 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840628586 0 0) 0x7fee681db350 con 0x7fee6813fc60 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:44.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee681dae30 con 0x7fee6813fc60 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392309189 0 0) 0x7fee680af5e0 con 0x7fee680b48e0 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:28:44.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.420+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee681db350 con 0x7fee680b48e0 2024-03-25T19:28:44.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1765054197 0 0) 0x7fee4c004eb0 con 0x7fee680b3f90 2024-03-25T19:28:44.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee680af5e0 con 0x7fee680b3f90 2024-03-25T19:28:44.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee64001a50 con 0x7fee680b3f90 2024-03-25T19:28:44.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:44.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 10 monclient(hunting): dump: 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3742552388 0 0) 0x7fee681db350 con 0x7fee680b48e0 2024-03-25T19:28:44.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.427+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee4c004eb0 con 0x7fee680b48e0 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3322842475 0 0) 0x7fee681dae30 con 0x7fee6813fc60 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee681db350 con 0x7fee6813fc60 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee60001c30 con 0x7fee6813fc60 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 10 monclient(hunting): dump: 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:44.430 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee58001b40 con 0x7fee680b48e0 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.428+0000 7fee5e7fc640 10 monclient(hunting): dump: 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:44.431 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:44.432 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:44.432 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:44.432 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1573149778 0 0) 0x7fee680af5e0 con 0x7fee680b3f90 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 >> v1:172.21.15.159:6789/0 conn(0x7fee680b48e0 legacy=0x7fee681d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 >> v1:172.21.15.134:6789/0 conn(0x7fee6813fc60 legacy=0x7fee68139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 1 monclient: found mon.c 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee681dab50 con 0x7fee680b3f90 2024-03-25T19:28:44.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.432+0000 7fee5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 5 monclient: authenticate success, global_id 4211 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 10 monclient: _renew_subs 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee681449c0 con 0x7fee680b3f90 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 10 monclient: _renew_subs 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee681d75d0 con 0x7fee680b3f90 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.433+0000 7fee6eb9e640 1 librados: init done 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.434+0000 7fee53fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.434+0000 7fee53fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.434+0000 7fee53fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:44.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.434+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee48005080 con 0x7fee680b3f90 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.459+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee640047e0 con 0x7fee680b3f90 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.459+0000 7fee5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.459+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee64004af0 con 0x7fee680b3f90 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.459+0000 7fee5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.459+0000 7fee5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.459+0000 7fee5e7fc640 10 monclient: dump: 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:44.459 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:44.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.459+0000 7fee5ffff640 4 set_mon_vals no callback set 2024-03-25T19:28:44.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.460+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fee6401aac0 con 0x7fee680b3f90 2024-03-25T19:28:44.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.460+0000 7fee6d39b640 10 monclient: get_auth_request con 0x7fee4c06e0a0 auth_method 0 2024-03-25T19:28:44.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.460+0000 7fee5e7fc640 10 client.4211.objecter ms_handle_connect 0x7fee4c06e0a0 2024-03-25T19:28:44.526 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:16.587 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.691 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:16.732 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.694 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:16.752 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fee64054790 con 0x7fee680b3f90 2024-03-25T19:28:44.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 10 client.4211.objecter ms_dispatch 0x7fee68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:44.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 3 client.4211.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:44.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 3 client.4211.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:44.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 20 client.4211.objecter dump_active .. 0 homeless 2024-03-25T19:28:44.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee640500d0 con 0x7fee680b3f90 2024-03-25T19:28:44.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:44.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.717+0000 7fee5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:44.763 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:16.826 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.846 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:16.895 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.897+0000 7fee53fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:44.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.897+0000 7fee53fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:44.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.897+0000 7fee53fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:28:44.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.897+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee48005330 con 0x7fee680b3f90 2024-03-25T19:28:44.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.913+0000 7fee5e7fc640 1 -- 172.21.15.134:0/122092160 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fee640521d0 con 0x7fee680b3f90 2024-03-25T19:28:44.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.913+0000 7fee5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:44.913 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.913+0000 7fee5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:44.913 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:28:44.913 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:44.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 10 librados: watch_flush enter 2024-03-25T19:28:44.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 10 librados: watch_flush exit 2024-03-25T19:28:44.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 20 client.4211.objecter shutdown clearing up homeless session... 2024-03-25T19:28:44.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 10 client.4211.objecter successfully canceled tick 2024-03-25T19:28:44.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 >> v1:172.21.15.159:6800/1875874887 conn(0x7fee4c06e0a0 legacy=0x7fee4c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 10 monclient: shutdown 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 >> v1:172.21.15.134:6790/0 conn(0x7fee680b3f90 legacy=0x7fee681da440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 shutdown_connections 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 >> 172.21.15.134:0/122092160 conn(0x7fee680615c0 msgr2=0x7fee6813eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 shutdown_connections 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 1 -- 172.21.15.134:0/122092160 wait complete. 2024-03-25T19:28:44.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:44.915+0000 7fee53fff640 1 librados: shutdown 2024-03-25T19:28:44.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:44.996 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:17.052 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:44.999 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:17.057 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:49.815 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:21.876 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:49.819 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:21.880 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.017 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:22.075 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.021 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:22.079 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.116 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:22.179 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.120 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:22.183 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.225 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:22.274 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.229 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:22.278 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.343 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:22.401 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.347 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:22.405 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.423 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:22.465 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.427 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:22.469 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.656 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:22.705 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.660 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:22.709 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.869 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:22.925 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:50.873 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:22.929 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:51.736 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.858+0000 7f05a9a84640 10 monclient: get_monmap_and_config 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.858+0000 7f05a9a84640 10 monclient: build_initial_monmap 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.858+0000 7f05a9a84640 10 monclient: monmap: 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:28:51.859182+0000 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:28:51.859182+0000 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:51.858 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:51.859 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:51.859 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:51.859 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:51.859 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.858+0000 7f05a9a84640 1 Processor -- start 2024-03-25T19:28:51.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 1 -- start start 2024-03-25T19:28:51.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient: init 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient(hunting): picked mon.noname-b con 0x7f05a40b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient(hunting): picked mon.noname-a con 0x7f05a40b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient(hunting): picked mon.noname-c con 0x7f05a414be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40af940 con 0x7f05a40b1730 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40b0a70 con 0x7f05a414be90 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40b0d50 con 0x7f05a40b4d60 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.859+0000 7f05a9a84640 10 monclient(hunting): authenticate will time out at 5964.018426s 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.860+0000 7f05a2ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f05a40b1730 0x7f05a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39706/0 (socket says 172.21.15.134:39706) 2024-03-25T19:28:51.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.860+0000 7f05a2ffd640 1 -- 172.21.15.134:0/2741429666 learned_addr learned my addr 172.21.15.134:0/2741429666 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:51.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.862+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442483473 0 0) 0x7f05a40b0a70 con 0x7f05a414be90 2024-03-25T19:28:51.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.862+0000 7f05a27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:51.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.862+0000 7f05a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:51.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.862+0000 7f05a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:51.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.862+0000 7f05a27fc640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:28:51.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.862+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c003520 con 0x7f05a414be90 2024-03-25T19:28:51.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703730004 0 0) 0x7f05a40af940 con 0x7f05a40b1730 2024-03-25T19:28:51.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:51.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:51.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:51.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): my global_id is 4230 2024-03-25T19:28:51.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a40b0a70 con 0x7f05a40b1730 2024-03-25T19:28:51.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720209445 0 0) 0x7f05a40b0d50 con 0x7f05a40b4d60 2024-03-25T19:28:51.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:51.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:51.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:51.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:28:51.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.863+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a40af940 con 0x7f05a40b4d60 2024-03-25T19:28:51.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3601449876 0 0) 0x7f058c003520 con 0x7f05a414be90 2024-03-25T19:28:51.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 >> v1:172.21.15.134:6790/0 conn(0x7f05a40b4d60 legacy=0x7f05a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:51.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 >> v1:172.21.15.134:6789/0 conn(0x7f05a40b1730 legacy=0x7f05a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:51.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:51.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 1 monclient: found mon.noname-c 2024-03-25T19:28:51.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a40b1030 con 0x7f05a414be90 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f059c001a10 con 0x7f05a414be90 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a27fc640 10 monclient: dump: 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:51.868 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:51.869 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:51.869 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:51.869 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:51.869 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a9a84640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:28:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.866+0000 7f05a9a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:51.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f059c001c30 con 0x7f05a414be90 2024-03-25T19:28:51.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:51.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a27fc640 1 -- 172.21.15.134:0/2741429666 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f059c001fc0 con 0x7f05a414be90 2024-03-25T19:28:51.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a9a84640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:51.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a9a84640 4 set_mon_vals no callback set 2024-03-25T19:28:51.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:51.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.868+0000 7f05a27fc640 10 monclient: dump: 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.869+0000 7f05a9a84640 10 monclient: shutdown 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.869+0000 7f05a9a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.869+0000 7f05a9a84640 1 -- 172.21.15.134:0/2741429666 >> v1:172.21.15.159:6789/0 conn(0x7f05a414be90 legacy=0x7f05a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:51.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.869+0000 7f05a9a84640 1 -- 172.21.15.134:0/2741429666 shutdown_connections 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.869+0000 7f05a9a84640 1 -- 172.21.15.134:0/2741429666 >> 172.21.15.134:0/2741429666 conn(0x7f05a40617a0 msgr2=0x7f05a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.869+0000 7f05a9a84640 1 -- 172.21.15.134:0/2741429666 shutdown_connections 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.869+0000 7f05a9a84640 1 -- 172.21.15.134:0/2741429666 wait complete. 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 10 monclient: build_initial_monmap 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 10 monclient: monmap: 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:51.874 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 1 librados: starting msgr at 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 1 librados: starting objecter 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 1 Processor -- start 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 1 -- start start 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 1 librados: setting wanted keys 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 1 librados: calling monclient init 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.870+0000 7f05a9a84640 10 monclient: init 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 10 monclient(hunting): picked mon.noname-c con 0x7f05a40b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 10 monclient(hunting): picked mon.noname-b con 0x7f05a40b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 10 monclient(hunting): picked mon.noname-a con 0x7f05a414be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a413bd10 con 0x7f05a414be90 2024-03-25T19:28:51.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a41dadb0 con 0x7f05a40b4d60 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a4138950 con 0x7f05a40b1730 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a9a84640 10 monclient(hunting): authenticate will time out at 5964.029836s 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a3fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f05a414be90 0x7f05a41db2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39722/0 (socket says 172.21.15.134:39722) 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f05a3fff640 1 -- 172.21.15.134:0/1018933352 learned_addr learned my addr 172.21.15.134:0/1018933352 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f058bfff640 10 client.?.objecter ms_handle_connect 0x7f05a414be90 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.871+0000 7f058bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.872+0000 7f058bfff640 10 client.?.objecter ms_handle_connect 0x7f05a40b4d60 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.872+0000 7f058bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.872+0000 7f058bfff640 10 client.?.objecter ms_handle_connect 0x7f05a40b1730 2024-03-25T19:28:51.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.872+0000 7f058bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047492097 0 0) 0x7f05a41dadb0 con 0x7f05a40b4d60 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0590004eb0 con 0x7f05a40b4d60 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824269466 0 0) 0x7f05a413bd10 con 0x7f05a414be90 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:51.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 10 monclient(hunting): my global_id is 4233 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.874+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a41dadb0 con 0x7f05a414be90 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.875+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298183919 0 0) 0x7f05a4138950 con 0x7f05a40b1730 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.875+0000 7f058bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.875+0000 7f058bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.875+0000 7f058bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.875+0000 7f058bfff640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.875+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a413bd10 con 0x7f05a40b1730 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.878+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2760112177 0 0) 0x7f0590004eb0 con 0x7f05a40b4d60 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.878+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05a4138950 con 0x7f05a40b4d60 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.878+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0598001970 con 0x7f05a40b4d60 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.878+0000 7f058bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.878+0000 7f058bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.878+0000 7f058bfff640 10 monclient(hunting): dump: 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:51.881 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.878+0000 7f058bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.879+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2269883714 0 0) 0x7f05a413bd10 con 0x7f05a40b1730 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.879+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0590004eb0 con 0x7f05a40b1730 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.880+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0594001830 con 0x7f05a40b1730 2024-03-25T19:28:51.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.880+0000 7f058bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.880+0000 7f058bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.880+0000 7f058bfff640 10 monclient(hunting): dump: 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.880+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998050529 0 0) 0x7f05a41dadb0 con 0x7f05a414be90 2024-03-25T19:28:51.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.880+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0590005ed0 con 0x7f05a414be90 2024-03-25T19:28:51.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.881+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f059c001f40 con 0x7f05a414be90 2024-03-25T19:28:51.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.881+0000 7f058bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:51.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.881+0000 7f058bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:51.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.881+0000 7f058bfff640 10 monclient(hunting): dump: 2024-03-25T19:28:51.886 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:51.886 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:51.886 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 30862385 0 0) 0x7f05a4138950 con 0x7f05a40b4d60 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 >> v1:172.21.15.134:6790/0 conn(0x7f05a40b1730 legacy=0x7f05a4138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 >> v1:172.21.15.134:6789/0 conn(0x7f05a414be90 legacy=0x7f05a41db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 1 monclient: found mon.b 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:51.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a413ab60 con 0x7f05a40b4d60 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 10 monclient: _finish_auth 0 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f058bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 10 monclient: _renew_subs 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a41db9e0 con 0x7f05a40b4d60 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 10 monclient: _renew_subs 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a4139890 con 0x7f05a40b4d60 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.882+0000 7f05a9a84640 1 librados: init done 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.884+0000 7f05a9a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.884+0000 7f05a9a84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.884+0000 7f05a9a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:51.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.884+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a413acf0 con 0x7f05a40b4d60 2024-03-25T19:28:51.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.905+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05980047e0 con 0x7f05a40b4d60 2024-03-25T19:28:51.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.905+0000 7f058bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:51.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.905+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0598004af0 con 0x7f05a40b4d60 2024-03-25T19:28:51.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.905+0000 7f058bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:51.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.905+0000 7f058bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:51.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.905+0000 7f058bfff640 10 monclient: dump: 2024-03-25T19:28:51.906 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:51.907 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:51.907 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.907 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:51.907 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:51.907 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:51.907 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:51.908 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:51.908 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:51.908 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:51.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.905+0000 7f05a1ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:51.910 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.906+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f059801ab00 con 0x7f05a40b4d60 2024-03-25T19:28:51.910 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.907+0000 7f05a37fe640 10 monclient: get_auth_request con 0x7f059006e3e0 auth_method 0 2024-03-25T19:28:51.910 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.907+0000 7f058bfff640 10 client.4219.objecter ms_handle_connect 0x7f059006e3e0 2024-03-25T19:28:51.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.909+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f05980547e0 con 0x7f05a40b4d60 2024-03-25T19:28:51.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.909+0000 7f058bfff640 10 client.4219.objecter ms_dispatch 0x7f05a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:51.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.909+0000 7f058bfff640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:51.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.909+0000 7f058bfff640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:51.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:51.909+0000 7f058bfff640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T19:28:52.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.171+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0598050120 con 0x7f05a40b4d60 2024-03-25T19:28:52.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.171+0000 7f058bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:52.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.171+0000 7f058bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:52.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.391+0000 7f05a9a84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:52.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.391+0000 7f05a9a84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:52.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.391+0000 7f05a9a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:52.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.391+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05a40627b0 con 0x7f05a40b4d60 2024-03-25T19:28:52.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.414+0000 7f058bfff640 1 -- 172.21.15.134:0/1018933352 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f0598052220 con 0x7f05a40b4d60 2024-03-25T19:28:52.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.415+0000 7f058bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:52.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.415+0000 7f058bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:52.415 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:28:52.415 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:52.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.416+0000 7f05a9a84640 10 librados: watch_flush enter 2024-03-25T19:28:52.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 10 librados: watch_flush exit 2024-03-25T19:28:52.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T19:28:52.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 10 client.4219.objecter successfully canceled tick 2024-03-25T19:28:52.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 >> v1:172.21.15.159:6800/1875874887 conn(0x7f059006e3e0 legacy=0x7f05900708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 10 monclient: shutdown 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 >> v1:172.21.15.159:6789/0 conn(0x7f05a40b4d60 legacy=0x7f05a413a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 shutdown_connections 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 >> 172.21.15.134:0/1018933352 conn(0x7f05a40617a0 msgr2=0x7f05a413c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 shutdown_connections 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 1 -- 172.21.15.134:0/1018933352 wait complete. 2024-03-25T19:28:52.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:52.417+0000 7f05a9a84640 1 librados: shutdown 2024-03-25T19:28:52.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:53.828 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:25.889 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:54.385 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:26.447 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:54.419 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:26.468 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:54.425 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:26.483 35434== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:54.657 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:26.715 35435== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:54.682 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:26.731 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:55.025 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:27.066 35444== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:55.230 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:27.286 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:59.235 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:59.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.355+0000 7f75c1e8f640 10 monclient: get_monmap_and_config 2024-03-25T19:28:59.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.355+0000 7f75c1e8f640 10 monclient: build_initial_monmap 2024-03-25T19:28:59.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.355+0000 7f75c1e8f640 10 monclient: monmap: 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:28:59.356477+0000 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:28:59.356477+0000 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:28:59.356 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.356+0000 7f75c1e8f640 1 Processor -- start 2024-03-25T19:28:59.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.356+0000 7f75c1e8f640 1 -- start start 2024-03-25T19:28:59.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.356+0000 7f75c1e8f640 10 monclient: init 2024-03-25T19:28:59.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:59.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f75bc0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 10 monclient(hunting): picked mon.noname-c con 0x7f75bc14be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0af940 con 0x7f75bc0b4d60 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0b0a70 con 0x7f75bc14be90 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0b0d50 con 0x7f75bc0b1730 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.357+0000 7f75c1e8f640 10 monclient(hunting): authenticate will time out at 5971.516018s 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.358+0000 7f75bbfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f75bc14be90 0x7f75bc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58644/0 (socket says 172.21.15.134:58644) 2024-03-25T19:28:59.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.358+0000 7f75bbfff640 1 -- 172.21.15.134:0/2633458862 learned_addr learned my addr 172.21.15.134:0/2633458862 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:59.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.362+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625589214 0 0) 0x7f75bc0b0a70 con 0x7f75bc14be90 2024-03-25T19:28:59.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.362+0000 7f75ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.362+0000 7f75ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.362+0000 7f75ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.362+0000 7f75ba7fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:28:59.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.362+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a4003520 con 0x7f75bc14be90 2024-03-25T19:28:59.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.367+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260285626 0 0) 0x7f75bc0b0d50 con 0x7f75bc0b1730 2024-03-25T19:28:59.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.367+0000 7f75ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.367+0000 7f75ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.367+0000 7f75ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.367+0000 7f75ba7fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:28:59.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.367+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc0b0a70 con 0x7f75bc0b1730 2024-03-25T19:28:59.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2171880207 0 0) 0x7f75a4003520 con 0x7f75bc14be90 2024-03-25T19:28:59.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 >> v1:172.21.15.134:6790/0 conn(0x7f75bc0b1730 legacy=0x7f75bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 >> v1:172.21.15.134:6789/0 conn(0x7f75bc0b4d60 legacy=0x7f75bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 1 monclient: found mon.noname-c 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0b1030 con 0x7f75bc14be90 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.369+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f75b4001a10 con 0x7f75bc14be90 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.370+0000 7f75ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.370+0000 7f75ba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.370+0000 7f75ba7fc640 10 monclient: dump: 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:59.371 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.370+0000 7f75c1e8f640 5 monclient: authenticate success, global_id 4222 2024-03-25T19:28:59.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.370+0000 7f75c1e8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b4001b70 con 0x7f75bc14be90 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75ba7fc640 1 -- 172.21.15.134:0/2633458862 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f75b4001f80 con 0x7f75bc14be90 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75ba7fc640 10 monclient: dump: 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:59.377 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:59.378 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:59.378 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:59.378 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75c1e8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:59.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.377+0000 7f75c1e8f640 4 set_mon_vals no callback set 2024-03-25T19:28:59.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.378+0000 7f75c1e8f640 10 monclient: shutdown 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.378+0000 7f75c1e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.378+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2633458862 >> v1:172.21.15.159:6789/0 conn(0x7f75bc14be90 legacy=0x7f75bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.378+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2633458862 shutdown_connections 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.378+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2633458862 >> 172.21.15.134:0/2633458862 conn(0x7f75bc0617a0 msgr2=0x7f75bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.378+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2633458862 shutdown_connections 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.378+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2633458862 wait complete. 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.379+0000 7f75c1e8f640 10 monclient: build_initial_monmap 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.379+0000 7f75c1e8f640 10 monclient: monmap: 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:59.380 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.379+0000 7f75c1e8f640 1 librados: starting msgr at 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.379+0000 7f75c1e8f640 1 librados: starting objecter 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.379+0000 7f75c1e8f640 1 Processor -- start 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.380+0000 7f75c1e8f640 1 -- start start 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.380+0000 7f75c1e8f640 1 librados: setting wanted keys 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.380+0000 7f75c1e8f640 1 librados: calling monclient init 2024-03-25T19:28:59.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.380+0000 7f75c1e8f640 10 monclient: init 2024-03-25T19:28:59.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.380+0000 7f75c1e8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:59.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.380+0000 7f75c1e8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:59.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.380+0000 7f75c1e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:28:59.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75c1e8f640 10 monclient(hunting): picked mon.noname-c con 0x7f75bc0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75c1e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f75bc14be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75c1e8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc135640 con 0x7f75bc0b1730 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75c1e8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc137b70 con 0x7f75bc14be90 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75c1e8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc137470 con 0x7f75bc0b4d60 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75c1e8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75c1e8f640 10 monclient(hunting): authenticate will time out at 5971.539984s 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75bb7fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f75bc0b1730 0x7f75bc142dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39740/0 (socket says 172.21.15.134:39740) 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75bb7fe640 1 -- 172.21.15.134:0/2905163819 learned_addr learned my addr 172.21.15.134:0/2905163819 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75a3fff640 10 client.?.objecter ms_handle_connect 0x7f75bc0b1730 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.381+0000 7f75a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.382+0000 7f75a3fff640 10 client.?.objecter ms_handle_connect 0x7f75bc0b4d60 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.382+0000 7f75a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.382+0000 7f75a3fff640 10 client.?.objecter ms_handle_connect 0x7f75bc14be90 2024-03-25T19:28:59.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.382+0000 7f75a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473113615 0 0) 0x7f75bc135640 con 0x7f75bc0b1730 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): my global_id is 4239 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a8004eb0 con 0x7f75bc0b1730 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873909724 0 0) 0x7f75bc137470 con 0x7f75bc0b4d60 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc135640 con 0x7f75bc0b4d60 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959264053 0 0) 0x7f75bc137b70 con 0x7f75bc14be90 2024-03-25T19:28:59.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:28:59.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.385+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc137470 con 0x7f75bc14be90 2024-03-25T19:28:59.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1941030160 0 0) 0x7f75bc137470 con 0x7f75bc14be90 2024-03-25T19:28:59.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75bc137b70 con 0x7f75bc14be90 2024-03-25T19:28:59.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f75b4001e80 con 0x7f75bc14be90 2024-03-25T19:28:59.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 10 monclient(hunting): dump: 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2068738925 0 0) 0x7f75a8004eb0 con 0x7f75bc0b1730 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75bc137470 con 0x7f75bc0b1730 2024-03-25T19:28:59.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f75ac001a30 con 0x7f75bc0b1730 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.391+0000 7f75a3fff640 10 monclient(hunting): dump: 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.392+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 774757933 0 0) 0x7f75bc135640 con 0x7f75bc0b4d60 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.392+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a8004eb0 con 0x7f75bc0b4d60 2024-03-25T19:28:59.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.392+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f75b0001930 con 0x7f75bc0b4d60 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.392+0000 7f75a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.392+0000 7f75a3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.392+0000 7f75a3fff640 10 monclient(hunting): dump: 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:59.395 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1498042091 0 0) 0x7f75bc137b70 con 0x7f75bc14be90 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 >> v1:172.21.15.134:6790/0 conn(0x7f75bc0b4d60 legacy=0x7f75bc138050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 >> v1:172.21.15.134:6789/0 conn(0x7f75bc0b1730 legacy=0x7f75bc142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 1 monclient: found mon.b 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc137750 con 0x7f75bc14be90 2024-03-25T19:28:59.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 5 monclient: authenticate success, global_id 4225 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 10 monclient: _renew_subs 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc1dc0a0 con 0x7f75bc14be90 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 10 monclient: _renew_subs 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc138760 con 0x7f75bc14be90 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.396+0000 7f75c1e8f640 1 librados: init done 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.397+0000 7f75c1e8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.397+0000 7f75c1e8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.397+0000 7f75c1e8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:28:59.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.397+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc1401f0 con 0x7f75bc14be90 2024-03-25T19:28:59.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.606+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b4001560 con 0x7f75bc14be90 2024-03-25T19:28:59.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.606+0000 7f75a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:59.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.606+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f75b4004e10 con 0x7f75bc14be90 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.606+0000 7f75a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.606+0000 7f75a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.606+0000 7f75a3fff640 10 monclient: dump: 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:28:59.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.606+0000 7f75b9ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:59.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.748+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f75b401ade0 con 0x7f75bc14be90 2024-03-25T19:28:59.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.749+0000 7f75bb7fe640 10 monclient: get_auth_request con 0x7f75a806e0b0 auth_method 0 2024-03-25T19:28:59.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.749+0000 7f75a3fff640 10 client.4225.objecter ms_handle_connect 0x7f75a806e0b0 2024-03-25T19:28:59.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.752+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f75b40013a0 con 0x7f75bc14be90 2024-03-25T19:28:59.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.752+0000 7f75a3fff640 10 client.4225.objecter ms_dispatch 0x7f75bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:59.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.752+0000 7f75a3fff640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:59.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.752+0000 7f75a3fff640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:59.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.753+0000 7f75a3fff640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-25T19:28:59.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.756+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75b4050200 con 0x7f75bc14be90 2024-03-25T19:28:59.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.756+0000 7f75a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:28:59.756+0000 7f75a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:00.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.104+0000 7f75c1e8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:00.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.104+0000 7f75c1e8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:00.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.104+0000 7f75c1e8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:00.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.104+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75bc13feb0 con 0x7f75bc14be90 2024-03-25T19:29:00.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.130+0000 7f75a3fff640 1 -- 172.21.15.134:0/2905163819 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f75b4052300 con 0x7f75bc14be90 2024-03-25T19:29:00.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.130+0000 7f75a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:00.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.130+0000 7f75a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:00.130 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:00.130 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:00.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.133+0000 7f75c1e8f640 10 librados: watch_flush enter 2024-03-25T19:29:00.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.133+0000 7f75c1e8f640 10 librados: watch_flush exit 2024-03-25T19:29:00.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.133+0000 7f75c1e8f640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-25T19:29:00.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.133+0000 7f75c1e8f640 10 client.4225.objecter successfully canceled tick 2024-03-25T19:29:00.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 >> v1:172.21.15.159:6800/1875874887 conn(0x7f75a806e0b0 legacy=0x7f75a8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 10 monclient: shutdown 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 >> v1:172.21.15.159:6789/0 conn(0x7f75bc14be90 legacy=0x7f75bc1dc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 shutdown_connections 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 >> 172.21.15.134:0/2905163819 conn(0x7f75bc0617a0 msgr2=0x7f75bc0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 shutdown_connections 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 1 -- 172.21.15.134:0/2905163819 wait complete. 2024-03-25T19:29:00.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:00.134+0000 7f75c1e8f640 1 librados: shutdown 2024-03-25T19:29:00.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:02.507 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-25T19:29:02.505+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:29:02.859 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-25T19:29:02.855+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:29:02.911 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-25T19:29:02.909+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:29:03.003 INFO:tasks.ceph.osd.2.smithi134.stderr:2024-03-25T19:29:03.001+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:29:03.008 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-25T19:29:03.004+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:29:03.321 INFO:tasks.ceph.osd.0.smithi134.stderr:2024-03-25T19:29:03.312+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:29:03.740 INFO:tasks.ceph.osd.1.smithi134.stderr:2024-03-25T19:29:03.736+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:29:03.926 INFO:tasks.ceph.osd.3.smithi134.stderr:2024-03-25T19:29:03.924+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:29:06.955 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:07.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.073+0000 7f32390e4640 10 monclient: get_monmap_and_config 2024-03-25T19:29:07.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.073+0000 7f32390e4640 10 monclient: build_initial_monmap 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.073+0000 7f32390e4640 10 monclient: monmap: 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:29:07.074983+0000 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:29:07.074983+0000 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:29:07.074 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.074+0000 7f32390e4640 1 Processor -- start 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.074+0000 7f32390e4640 1 -- start start 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.074+0000 7f32390e4640 10 monclient: init 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 10 monclient(hunting): picked mon.noname-b con 0x7f32340b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 10 monclient(hunting): picked mon.noname-c con 0x7f32340b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 10 monclient(hunting): picked mon.noname-a con 0x7f323414be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32340af940 con 0x7f323414be90 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32340b0a70 con 0x7f32340b1730 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32340b0d50 con 0x7f32340b4d60 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:07.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.075+0000 7f32390e4640 10 monclient(hunting): authenticate will time out at 5979.234383s 2024-03-25T19:29:07.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.082+0000 7f3232d9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f32340b4d60 0x7f323413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:45444/0 (socket says 172.21.15.134:45444) 2024-03-25T19:29:07.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.082+0000 7f3232d9d640 1 -- 172.21.15.134:0/438136745 learned_addr learned my addr 172.21.15.134:0/438136745 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:07.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.097+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495693474 0 0) 0x7f32340b0d50 con 0x7f32340b4d60 2024-03-25T19:29:07.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.097+0000 7f323259c640 10 monclient(hunting): _negotiate 2024-03-25T19:29:07.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.097+0000 7f323259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:07.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.097+0000 7f323259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:07.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.097+0000 7f323259c640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:29:07.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.097+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3220003520 con 0x7f32340b4d60 2024-03-25T19:29:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.104+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222379937 0 0) 0x7f32340b0a70 con 0x7f32340b1730 2024-03-25T19:29:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.104+0000 7f323259c640 10 monclient(hunting): _negotiate 2024-03-25T19:29:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.104+0000 7f323259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.104+0000 7f323259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.104+0000 7f323259c640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:29:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.104+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32340b0d50 con 0x7f32340b1730 2024-03-25T19:29:07.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1766574093 0 0) 0x7f3220003520 con 0x7f32340b4d60 2024-03-25T19:29:07.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 >> v1:172.21.15.159:6789/0 conn(0x7f32340b1730 legacy=0x7f32340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:07.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 >> v1:172.21.15.134:6789/0 conn(0x7f323414be90 legacy=0x7f323414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:07.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 10 monclient: _finish_hunting 0 2024-03-25T19:29:07.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 1 monclient: found mon.noname-b 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32340b1030 con 0x7f32340b4d60 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 10 monclient: _finish_auth 0 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 10 monclient: _check_auth_tickets 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3224001690 con 0x7f32340b4d60 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f323259c640 10 monclient: dump: 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:07.117 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f32390e4640 5 monclient: authenticate success, global_id 4250 2024-03-25T19:29:07.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.116+0000 7f32390e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:07.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.132+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32240017f0 con 0x7f32340b4d60 2024-03-25T19:29:07.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.132+0000 7f323259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.132+0000 7f323259c640 1 -- 172.21.15.134:0/438136745 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3224001c00 con 0x7f32340b4d60 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 4 set_mon_vals no callback set 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f323259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f323259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f323259c640 10 monclient: dump: 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:07.133 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 10 monclient: shutdown 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 1 -- 172.21.15.134:0/438136745 >> v1:172.21.15.134:6790/0 conn(0x7f32340b4d60 legacy=0x7f323413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 1 -- 172.21.15.134:0/438136745 shutdown_connections 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 1 -- 172.21.15.134:0/438136745 >> 172.21.15.134:0/438136745 conn(0x7f32340617a0 msgr2=0x7f3234061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 1 -- 172.21.15.134:0/438136745 shutdown_connections 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.133+0000 7f32390e4640 1 -- 172.21.15.134:0/438136745 wait complete. 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.134+0000 7f32390e4640 10 monclient: build_initial_monmap 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.134+0000 7f32390e4640 10 monclient: monmap: 2024-03-25T19:29:07.135 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.135+0000 7f32390e4640 1 librados: starting msgr at 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.135+0000 7f32390e4640 1 librados: starting objecter 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.135+0000 7f32390e4640 1 Processor -- start 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.135+0000 7f32390e4640 1 -- start start 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.135+0000 7f32390e4640 1 librados: setting wanted keys 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.135+0000 7f32390e4640 1 librados: calling monclient init 2024-03-25T19:29:07.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.135+0000 7f32390e4640 10 monclient: init 2024-03-25T19:29:07.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:07.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 10 monclient(hunting): picked mon.noname-b con 0x7f32340b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 10 monclient(hunting): picked mon.noname-c con 0x7f32340b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 10 monclient(hunting): picked mon.noname-a con 0x7f323414be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32341361d0 con 0x7f323414be90 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3234136c40 con 0x7f32340b1730 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3234136f20 con 0x7f32340b4d60 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:07.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.136+0000 7f32390e4640 10 monclient(hunting): authenticate will time out at 5979.294950s 2024-03-25T19:29:07.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.149+0000 7f323359e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f323414be90 0x7f32341e33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:47518/0 (socket says 172.21.15.134:47518) 2024-03-25T19:29:07.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.149+0000 7f323359e640 1 -- 172.21.15.134:0/1177548879 learned_addr learned my addr 172.21.15.134:0/1177548879 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:07.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.150+0000 7f322b7fe640 10 client.?.objecter ms_handle_connect 0x7f323414be90 2024-03-25T19:29:07.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.150+0000 7f322b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:07.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.163+0000 7f322b7fe640 10 client.?.objecter ms_handle_connect 0x7f32340b1730 2024-03-25T19:29:07.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.163+0000 7f322b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:07.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.165+0000 7f322b7fe640 10 client.?.objecter ms_handle_connect 0x7f32340b4d60 2024-03-25T19:29:07.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.165+0000 7f322b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:07.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.167+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199486738 0 0) 0x7f32341361d0 con 0x7f323414be90 2024-03-25T19:29:07.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.167+0000 7f322b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:07.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.167+0000 7f322b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:07.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.167+0000 7f322b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:07.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.167+0000 7f322b7fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:29:07.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.167+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3214004eb0 con 0x7f323414be90 2024-03-25T19:29:07.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.171+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465664250 0 0) 0x7f3234136f20 con 0x7f32340b4d60 2024-03-25T19:29:07.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.171+0000 7f322b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:07.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.171+0000 7f322b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:07.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.171+0000 7f322b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:07.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.171+0000 7f322b7fe640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:29:07.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.171+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32341361d0 con 0x7f32340b4d60 2024-03-25T19:29:07.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.175+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182772960 0 0) 0x7f3234136c40 con 0x7f32340b1730 2024-03-25T19:29:07.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.175+0000 7f322b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:07.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.175+0000 7f322b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:07.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.175+0000 7f322b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:07.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.175+0000 7f322b7fe640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:29:07.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.175+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3234136f20 con 0x7f32340b1730 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.190+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2158640605 0 0) 0x7f3234136f20 con 0x7f32340b1730 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.190+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3234136c40 con 0x7f32340b1730 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.190+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3224001b00 con 0x7f32340b1730 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.190+0000 7f322b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.190+0000 7f322b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.190+0000 7f322b7fe640 10 monclient(hunting): dump: 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:07.190 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:07.191 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:07.191 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:07.191 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.190+0000 7f322b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.192+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3252600804 0 0) 0x7f3214004eb0 con 0x7f323414be90 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.192+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3234136f20 con 0x7f323414be90 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.192+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f322c001c30 con 0x7f323414be90 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.192+0000 7f322b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.192+0000 7f322b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.192+0000 7f322b7fe640 10 monclient(hunting): dump: 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:07.193 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.193+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4040263284 0 0) 0x7f32341361d0 con 0x7f32340b4d60 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.193+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3214004eb0 con 0x7f32340b4d60 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.193+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f321c001930 con 0x7f32340b4d60 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.193+0000 7f322b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:07.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.193+0000 7f322b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.193+0000 7f322b7fe640 10 monclient(hunting): dump: 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:07.195 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.199+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2191677669 0 0) 0x7f3234136c40 con 0x7f32340b1730 2024-03-25T19:29:07.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 >> v1:172.21.15.134:6790/0 conn(0x7f32340b4d60 legacy=0x7f323412cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:07.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 >> v1:172.21.15.134:6789/0 conn(0x7f323414be90 legacy=0x7f32341e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:07.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:07.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 1 monclient: found mon.b 2024-03-25T19:29:07.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:07.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234137200 con 0x7f32340b1730 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f322b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 10 monclient: _renew_subs 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3234135cb0 con 0x7f32340b1730 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 10 monclient: _renew_subs 2024-03-25T19:29:07.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:07.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f323413e930 con 0x7f32340b1730 2024-03-25T19:29:07.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.200+0000 7f32390e4640 1 librados: init done 2024-03-25T19:29:07.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.201+0000 7f32390e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:07.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.201+0000 7f32390e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:07.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.201+0000 7f32390e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:07.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.201+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32341401f0 con 0x7f32340b1730 2024-03-25T19:29:07.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.468+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3224004a90 con 0x7f32340b1730 2024-03-25T19:29:07.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.468+0000 7f322b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:07.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.468+0000 7f3231d9b640 4 set_mon_vals no callback set 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.477+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3224004da0 con 0x7f32340b1730 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.477+0000 7f322b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.477+0000 7f322b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.477+0000 7f322b7fe640 10 monclient: dump: 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:07.477 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:07.478 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:07.478 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:07.478 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:07.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.478+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f322401ad70 con 0x7f32340b1730 2024-03-25T19:29:07.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.478+0000 7f322bfff640 10 monclient: get_auth_request con 0x7f321406e0b0 auth_method 0 2024-03-25T19:29:07.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.479+0000 7f322b7fe640 10 client.4255.objecter ms_handle_connect 0x7f321406e0b0 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.484+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f32240548c0 con 0x7f32340b1730 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.484+0000 7f322b7fe640 10 client.4255.objecter ms_dispatch 0x7f3234000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.484+0000 7f322b7fe640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.484+0000 7f322b7fe640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.484+0000 7f322b7fe640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.485+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32240503d0 con 0x7f32340b1730 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.485+0000 7f322b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:07.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.485+0000 7f322b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:07.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.723+0000 7f32390e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:07.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.723+0000 7f32390e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:07.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.723+0000 7f32390e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:07.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.723+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32340afbd0 con 0x7f32340b1730 2024-03-25T19:29:07.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.746+0000 7f322b7fe640 1 -- 172.21.15.134:0/1177548879 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f3224052300 con 0x7f32340b1730 2024-03-25T19:29:07.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.746+0000 7f322b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:07.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.746+0000 7f322b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:07.746 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:07.746 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:07.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 10 librados: watch_flush enter 2024-03-25T19:29:07.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 10 librados: watch_flush exit 2024-03-25T19:29:07.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-25T19:29:07.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 10 client.4255.objecter successfully canceled tick 2024-03-25T19:29:07.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 >> v1:172.21.15.159:6800/1875874887 conn(0x7f321406e0b0 legacy=0x7f3214070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 10 monclient: shutdown 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 >> v1:172.21.15.159:6789/0 conn(0x7f32340b1730 legacy=0x7f323412e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 shutdown_connections 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 >> 172.21.15.134:0/1177548879 conn(0x7f32340617a0 msgr2=0x7f32340b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 shutdown_connections 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 1 -- 172.21.15.134:0/1177548879 wait complete. 2024-03-25T19:29:07.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:07.748+0000 7f32390e4640 1 librados: shutdown 2024-03-25T19:29:07.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:14.567 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:14.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.667+0000 7feccdcfb640 10 monclient: get_monmap_and_config 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.667+0000 7feccdcfb640 10 monclient: build_initial_monmap 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.667+0000 7feccdcfb640 10 monclient: monmap: 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:29:14.668839+0000 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:29:14.668839+0000 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:29:14.668 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:14.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.668+0000 7feccdcfb640 1 Processor -- start 2024-03-25T19:29:14.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.669+0000 7feccdcfb640 1 -- start start 2024-03-25T19:29:14.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.669+0000 7feccdcfb640 10 monclient: init 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 10 monclient(hunting): picked mon.noname-a con 0x7fecc80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 10 monclient(hunting): picked mon.noname-c con 0x7fecc813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 10 monclient(hunting): picked mon.noname-b con 0x7fecc81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc8142520 con 0x7fecc80a98f0 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80b28a0 con 0x7fecc813f710 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80b2a30 con 0x7fecc81430c0 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:14.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.670+0000 7feccdcfb640 10 monclient(hunting): authenticate will time out at 5986.828942s 2024-03-25T19:29:14.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.675+0000 7fecc7fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fecc81430c0 0x7fecc814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:35594/0 (socket says 172.21.15.134:35594) 2024-03-25T19:29:14.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.675+0000 7fecc7fff640 1 -- 172.21.15.134:0/3336810686 learned_addr learned my addr 172.21.15.134:0/3336810686 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:14.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.683+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173782329 0 0) 0x7fecc80b2a30 con 0x7fecc81430c0 2024-03-25T19:29:14.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.683+0000 7fecc67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:14.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.683+0000 7fecc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:14.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.683+0000 7fecc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:14.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.683+0000 7fecc67fc640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:29:14.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.683+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0003520 con 0x7fecc81430c0 2024-03-25T19:29:14.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.693+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623815596 0 0) 0x7fecc8142520 con 0x7fecc80a98f0 2024-03-25T19:29:14.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.693+0000 7fecc67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:14.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.693+0000 7fecc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:14.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.693+0000 7fecc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:14.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.693+0000 7fecc67fc640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:29:14.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.693+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc80b2a30 con 0x7fecc80a98f0 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 84416296 0 0) 0x7fecb0003520 con 0x7fecc81430c0 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 >> v1:172.21.15.159:6789/0 conn(0x7fecc813f710 legacy=0x7fecc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 >> v1:172.21.15.134:6789/0 conn(0x7fecc80a98f0 legacy=0x7fecc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 1 monclient: found mon.noname-b 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc80b44c0 con 0x7fecc81430c0 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fecc0001a10 con 0x7fecc81430c0 2024-03-25T19:29:14.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7fecc67fc640 10 monclient: dump: 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:14.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7feccdcfb640 5 monclient: authenticate success, global_id 4256 2024-03-25T19:29:14.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.700+0000 7feccdcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc0001b70 con 0x7fecc81430c0 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7fecc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7fecc67fc640 1 -- 172.21.15.134:0/3336810686 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fecc0001f80 con 0x7fecc81430c0 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7fecc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7fecc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7fecc67fc640 10 monclient: dump: 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:14.709 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7feccdcfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:14.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.709+0000 7feccdcfb640 4 set_mon_vals no callback set 2024-03-25T19:29:14.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.710+0000 7feccdcfb640 10 monclient: shutdown 2024-03-25T19:29:14.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.710+0000 7feccdcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:14.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.710+0000 7feccdcfb640 1 -- 172.21.15.134:0/3336810686 >> v1:172.21.15.134:6790/0 conn(0x7fecc81430c0 legacy=0x7fecc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:14.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.710+0000 7feccdcfb640 1 -- 172.21.15.134:0/3336810686 shutdown_connections 2024-03-25T19:29:14.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.710+0000 7feccdcfb640 1 -- 172.21.15.134:0/3336810686 >> 172.21.15.134:0/3336810686 conn(0x7fecc80b7ea0 msgr2=0x7fecc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.710+0000 7feccdcfb640 1 -- 172.21.15.134:0/3336810686 shutdown_connections 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.710+0000 7feccdcfb640 1 -- 172.21.15.134:0/3336810686 wait complete. 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.711+0000 7feccdcfb640 10 monclient: build_initial_monmap 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.711+0000 7feccdcfb640 10 monclient: monmap: 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.711+0000 7feccdcfb640 1 librados: starting msgr at 2024-03-25T19:29:14.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.711+0000 7feccdcfb640 1 librados: starting objecter 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.711+0000 7feccdcfb640 1 Processor -- start 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 1 -- start start 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 1 librados: setting wanted keys 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 1 librados: calling monclient init 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient: init 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient(hunting): picked mon.noname-c con 0x7fecc80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient(hunting): picked mon.noname-b con 0x7fecc813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient(hunting): picked mon.noname-a con 0x7fecc81430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc81daca0 con 0x7fecc81430c0 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc8138ab0 con 0x7fecc813f710 2024-03-25T19:29:14.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80aba90 con 0x7fecc80a98f0 2024-03-25T19:29:14.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:14.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.712+0000 7feccdcfb640 10 monclient(hunting): authenticate will time out at 5986.871473s 2024-03-25T19:29:14.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.721+0000 7fecc7fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fecc81430c0 0x7fecc81d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:45622/0 (socket says 172.21.15.134:45622) 2024-03-25T19:29:14.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.721+0000 7fecc7fff640 1 -- 172.21.15.134:0/353862395 learned_addr learned my addr 172.21.15.134:0/353862395 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:14.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.723+0000 7fecaffff640 10 client.?.objecter ms_handle_connect 0x7fecc81430c0 2024-03-25T19:29:14.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.723+0000 7fecaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:14.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.730+0000 7fecaffff640 10 client.?.objecter ms_handle_connect 0x7fecc80a98f0 2024-03-25T19:29:14.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.730+0000 7fecaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.733+0000 7fecaffff640 10 client.?.objecter ms_handle_connect 0x7fecc813f710 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.733+0000 7fecaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.734+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602058448 0 0) 0x7fecc81daca0 con 0x7fecc81430c0 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.734+0000 7fecaffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.734+0000 7fecaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.734+0000 7fecaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.734+0000 7fecaffff640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:29:14.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.734+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb4004eb0 con 0x7fecc81430c0 2024-03-25T19:29:14.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.742+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421994612 0 0) 0x7fecc80aba90 con 0x7fecc80a98f0 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.742+0000 7fecaffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.742+0000 7fecaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.742+0000 7fecaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.742+0000 7fecaffff640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc81daca0 con 0x7fecc80a98f0 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882552183 0 0) 0x7fecb4004eb0 con 0x7fecc81430c0 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecc80aba90 con 0x7fecc81430c0 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fecc0001ec0 con 0x7fecc81430c0 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 10 monclient(hunting): dump: 2024-03-25T19:29:14.743 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:14.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.743+0000 7fecaffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:14.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.751+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2329780863 0 0) 0x7fecc81daca0 con 0x7fecc80a98f0 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.751+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecb4004eb0 con 0x7fecc80a98f0 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.751+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fecb8001930 con 0x7fecc80a98f0 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.751+0000 7fecaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.751+0000 7fecaffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.751+0000 7fecaffff640 10 monclient(hunting): dump: 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:14.751 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:14.752 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:14.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3280938021 0 0) 0x7fecc80aba90 con 0x7fecc81430c0 2024-03-25T19:29:14.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 >> v1:172.21.15.134:6790/0 conn(0x7fecc80a98f0 legacy=0x7fecc81383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:14.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 >> v1:172.21.15.159:6789/0 conn(0x7fecc813f710 legacy=0x7fecc81db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:14.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 1 monclient: found mon.a 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc81d41e0 con 0x7fecc81430c0 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 10 monclient: _finish_auth 0 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7fecaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 5 monclient: authenticate success, global_id 4275 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 10 monclient: _renew_subs 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc81db840 con 0x7fecc81430c0 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 10 monclient: _renew_subs 2024-03-25T19:29:14.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:14.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc8138f00 con 0x7fecc81430c0 2024-03-25T19:29:14.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.754+0000 7feccdcfb640 1 librados: init done 2024-03-25T19:29:14.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.755+0000 7feccdcfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:14.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.755+0000 7feccdcfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:14.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.755+0000 7feccdcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:14.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:14.755+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc81dae50 con 0x7fecc81430c0 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.066+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc0004e10 con 0x7fecc81430c0 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.066+0000 7fecaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.066+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fecc0002100 con 0x7fecc81430c0 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.066+0000 7fecaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.066+0000 7fecaffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.066+0000 7fecaffff640 10 monclient: dump: 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:15.066 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.066+0000 7fecc5ffb640 4 set_mon_vals no callback set 2024-03-25T19:29:15.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.067+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fecc001adf0 con 0x7fecc81430c0 2024-03-25T19:29:15.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.067+0000 7fecc77fe640 10 monclient: get_auth_request con 0x7fecb406de90 auth_method 0 2024-03-25T19:29:15.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.067+0000 7fecaffff640 10 client.4275.objecter ms_handle_connect 0x7fecb406de90 2024-03-25T19:29:15.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fecc0054980 con 0x7fecc81430c0 2024-03-25T19:29:15.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 10 client.4275.objecter ms_dispatch 0x7fecc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:15.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:15.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:15.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-25T19:29:15.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecc0050490 con 0x7fecc81430c0 2024-03-25T19:29:15.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:15.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.071+0000 7fecaffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:15.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.334+0000 7feccdcfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:15.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.334+0000 7feccdcfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:15.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.334+0000 7feccdcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:15.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.334+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecc80627b0 con 0x7fecc81430c0 2024-03-25T19:29:15.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.392+0000 7fecaffff640 1 -- 172.21.15.134:0/353862395 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fecc00523c0 con 0x7fecc81430c0 2024-03-25T19:29:15.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.392+0000 7fecaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:15.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.392+0000 7fecaffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:15.393 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:15.393 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 10 librados: watch_flush enter 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 10 librados: watch_flush exit 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 10 client.4275.objecter successfully canceled tick 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 >> v1:172.21.15.159:6800/1875874887 conn(0x7fecb406de90 legacy=0x7fecb4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 10 monclient: shutdown 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 >> v1:172.21.15.134:6789/0 conn(0x7fecc81430c0 legacy=0x7fecc81d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 shutdown_connections 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 >> 172.21.15.134:0/353862395 conn(0x7fecc80b7ea0 msgr2=0x7fecc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.395+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 shutdown_connections 2024-03-25T19:29:15.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.396+0000 7feccdcfb640 1 -- 172.21.15.134:0/353862395 wait complete. 2024-03-25T19:29:15.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:15.396+0000 7feccdcfb640 1 librados: shutdown 2024-03-25T19:29:15.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:19.624 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:24.297 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:19.686 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:24.358 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:19.745 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:24.392 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:19.804 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:24.451 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:19.823 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:24.495 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:19.894 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:24.567 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:19.906 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:24.579 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:20.040 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:24.687 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:20.129 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:24.801 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:22.213 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:22.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.320+0000 7f2d6e92e640 10 monclient: get_monmap_and_config 2024-03-25T19:29:22.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.320+0000 7f2d6e92e640 10 monclient: build_initial_monmap 2024-03-25T19:29:22.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.320+0000 7f2d6e92e640 10 monclient: monmap: 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:29:22.322009+0000 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:29:22.322009+0000 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:29:22.321 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.321+0000 7f2d6e92e640 1 Processor -- start 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 1 -- start start 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient: init 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d680a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d6813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d681430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d68142520 con 0x7f2d681430c0 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680b28a0 con 0x7f2d6813f710 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680b2a30 con 0x7f2d680a98f0 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:22.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.322+0000 7f2d6e92e640 10 monclient(hunting): authenticate will time out at 5994.481509s 2024-03-25T19:29:22.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.328+0000 7f2d67fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f2d680a98f0 0x7f2d680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46548/0 (socket says 172.21.15.134:46548) 2024-03-25T19:29:22.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.328+0000 7f2d67fff640 1 -- 172.21.15.134:0/3220617761 learned_addr learned my addr 172.21.15.134:0/3220617761 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:22.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.334+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620772654 0 0) 0x7f2d68142520 con 0x7f2d681430c0 2024-03-25T19:29:22.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.334+0000 7f2d66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:22.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.334+0000 7f2d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:22.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.334+0000 7f2d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:22.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.334+0000 7f2d66ffd640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:29:22.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.334+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d50003520 con 0x7f2d681430c0 2024-03-25T19:29:22.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.350+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918408738 0 0) 0x7f2d680b2a30 con 0x7f2d680a98f0 2024-03-25T19:29:22.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.350+0000 7f2d66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:22.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.350+0000 7f2d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:22.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.350+0000 7f2d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:22.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.350+0000 7f2d66ffd640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:29:22.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.350+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d68142520 con 0x7f2d680a98f0 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.351+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3172142975 0 0) 0x7f2d50003520 con 0x7f2d681430c0 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 >> v1:172.21.15.134:6790/0 conn(0x7f2d680a98f0 legacy=0x7f2d680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 >> v1:172.21.15.159:6789/0 conn(0x7f2d6813f710 legacy=0x7f2d681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 1 monclient: found mon.noname-a 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680b44c0 con 0x7f2d681430c0 2024-03-25T19:29:22.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2d60001a10 con 0x7f2d681430c0 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d6e92e640 5 monclient: authenticate success, global_id 4278 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d6e92e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d66ffd640 10 monclient: dump: 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:22.353 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:22.354 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:22.354 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:22.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.352+0000 7f2d6e92e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:22.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d60001c30 con 0x7f2d681430c0 2024-03-25T19:29:22.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:22.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d66ffd640 1 -- 172.21.15.134:0/3220617761 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2d60001fc0 con 0x7f2d681430c0 2024-03-25T19:29:22.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:22.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:22.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d66ffd640 10 monclient: dump: 2024-03-25T19:29:22.372 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d6e92e640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:22.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.372+0000 7f2d6e92e640 4 set_mon_vals no callback set 2024-03-25T19:29:22.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.373+0000 7f2d6e92e640 10 monclient: shutdown 2024-03-25T19:29:22.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.373+0000 7f2d6e92e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:22.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.373+0000 7f2d6e92e640 1 -- 172.21.15.134:0/3220617761 >> v1:172.21.15.134:6789/0 conn(0x7f2d681430c0 legacy=0x7f2d6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.373+0000 7f2d6e92e640 1 -- 172.21.15.134:0/3220617761 shutdown_connections 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.373+0000 7f2d6e92e640 1 -- 172.21.15.134:0/3220617761 >> 172.21.15.134:0/3220617761 conn(0x7f2d680b7ea0 msgr2=0x7f2d680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.373+0000 7f2d6e92e640 1 -- 172.21.15.134:0/3220617761 shutdown_connections 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.373+0000 7f2d6e92e640 1 -- 172.21.15.134:0/3220617761 wait complete. 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.374+0000 7f2d6e92e640 10 monclient: build_initial_monmap 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.374+0000 7f2d6e92e640 10 monclient: monmap: 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:29:22.375 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:29:22.376 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:22.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.375+0000 7f2d6e92e640 1 librados: starting msgr at 2024-03-25T19:29:22.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.375+0000 7f2d6e92e640 1 librados: starting objecter 2024-03-25T19:29:22.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.375+0000 7f2d6e92e640 1 Processor -- start 2024-03-25T19:29:22.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.375+0000 7f2d6e92e640 1 -- start start 2024-03-25T19:29:22.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.375+0000 7f2d6e92e640 1 librados: setting wanted keys 2024-03-25T19:29:22.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.375+0000 7f2d6e92e640 1 librados: calling monclient init 2024-03-25T19:29:22.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.375+0000 7f2d6e92e640 10 monclient: init 2024-03-25T19:29:22.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d680a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d6813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d681430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d68136e20 con 0x7f2d680a98f0 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680b7940 con 0x7f2d6813f710 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d681351a0 con 0x7f2d681430c0 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:22.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.376+0000 7f2d6e92e640 10 monclient(hunting): authenticate will time out at 5994.535301s 2024-03-25T19:29:22.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.384+0000 7f2d6cecb640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f2d681430c0 0x7f2d681e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46612/0 (socket says 172.21.15.134:46612) 2024-03-25T19:29:22.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.384+0000 7f2d6cecb640 1 -- 172.21.15.134:0/1360155346 learned_addr learned my addr 172.21.15.134:0/1360155346 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:22.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.386+0000 7f2d64ff9640 10 client.?.objecter ms_handle_connect 0x7f2d681430c0 2024-03-25T19:29:22.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.386+0000 7f2d64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:22.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.389+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980041068 0 0) 0x7f2d681351a0 con 0x7f2d681430c0 2024-03-25T19:29:22.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.389+0000 7f2d64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:22.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.389+0000 7f2d64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:22.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.389+0000 7f2d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:22.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.389+0000 7f2d64ff9640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:29:22.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.389+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54004eb0 con 0x7f2d681430c0 2024-03-25T19:29:22.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.390+0000 7f2d64ff9640 10 client.?.objecter ms_handle_connect 0x7f2d680a98f0 2024-03-25T19:29:22.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.390+0000 7f2d64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:22.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.392+0000 7f2d64ff9640 10 client.?.objecter ms_handle_connect 0x7f2d6813f710 2024-03-25T19:29:22.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.392+0000 7f2d64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.398+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1255826097 0 0) 0x7f2d54004eb0 con 0x7f2d681430c0 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.398+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d681351a0 con 0x7f2d681430c0 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.398+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2d60001f80 con 0x7f2d681430c0 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.398+0000 7f2d64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.398+0000 7f2d64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.398+0000 7f2d64ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:22.398 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:22.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.398+0000 7f2d64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:22.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.403+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585396754 0 0) 0x7f2d680b7940 con 0x7f2d6813f710 2024-03-25T19:29:22.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.403+0000 7f2d64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:22.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.403+0000 7f2d64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:22.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.403+0000 7f2d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:22.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.403+0000 7f2d64ff9640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:29:22.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.403+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54004eb0 con 0x7f2d6813f710 2024-03-25T19:29:22.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.409+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679076759 0 0) 0x7f2d68136e20 con 0x7f2d680a98f0 2024-03-25T19:29:22.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.409+0000 7f2d64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:22.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.409+0000 7f2d64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:22.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.409+0000 7f2d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:22.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.409+0000 7f2d64ff9640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:29:22.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.409+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d680b7940 con 0x7f2d680a98f0 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1717778285 0 0) 0x7f2d681351a0 con 0x7f2d681430c0 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 >> v1:172.21.15.159:6789/0 conn(0x7f2d6813f710 legacy=0x7f2d681db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 >> v1:172.21.15.134:6789/0 conn(0x7f2d680a98f0 legacy=0x7f2d681daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 1 monclient: found mon.c 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d68135330 con 0x7f2d681430c0 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d6e92e640 5 monclient: authenticate success, global_id 4265 2024-03-25T19:29:22.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d6e92e640 10 monclient: _renew_subs 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d6e92e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.415+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d68135560 con 0x7f2d681430c0 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 10 monclient: _renew_subs 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6813b490 con 0x7f2d681430c0 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 1 librados: init done 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:22.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:22.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:22.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.416+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d680b7ad0 con 0x7f2d681430c0 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.737+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d60003c20 con 0x7f2d681430c0 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.738+0000 7f2d64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.738+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2d60003ff0 con 0x7f2d681430c0 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.738+0000 7f2d64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.738+0000 7f2d64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.738+0000 7f2d64ff9640 10 monclient: dump: 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:22.738 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.738+0000 7f2d667fc640 4 set_mon_vals no callback set 2024-03-25T19:29:22.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.739+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f2d60004210 con 0x7f2d681430c0 2024-03-25T19:29:22.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.739+0000 7f2d67fff640 10 monclient: get_auth_request con 0x7f2d5406de80 auth_method 0 2024-03-25T19:29:22.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.739+0000 7f2d64ff9640 10 client.4265.objecter ms_handle_connect 0x7f2d5406de80 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.744+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f2d60054be0 con 0x7f2d681430c0 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.744+0000 7f2d64ff9640 10 client.4265.objecter ms_dispatch 0x7f2d68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.744+0000 7f2d64ff9640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.744+0000 7f2d64ff9640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.745+0000 7f2d64ff9640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.745+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d600506f0 con 0x7f2d681430c0 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.745+0000 7f2d64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:22.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:22.745+0000 7f2d64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:23.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.003+0000 7f2d6e92e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:23.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.003+0000 7f2d6e92e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:23.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.003+0000 7f2d6e92e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:23.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.003+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d680b7730 con 0x7f2d681430c0 2024-03-25T19:29:23.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.061+0000 7f2d64ff9640 1 -- 172.21.15.134:0/1360155346 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f2d60052620 con 0x7f2d681430c0 2024-03-25T19:29:23.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.061+0000 7f2d64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:23.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.061+0000 7f2d64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:23.062 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:23.062 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:23.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 10 librados: watch_flush enter 2024-03-25T19:29:23.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 10 librados: watch_flush exit 2024-03-25T19:29:23.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-25T19:29:23.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 10 client.4265.objecter successfully canceled tick 2024-03-25T19:29:23.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 >> v1:172.21.15.159:6800/1875874887 conn(0x7f2d5406de80 legacy=0x7f2d54070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:23.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 10 monclient: shutdown 2024-03-25T19:29:23.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:23.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 >> v1:172.21.15.134:6790/0 conn(0x7f2d681430c0 legacy=0x7f2d681e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:23.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 shutdown_connections 2024-03-25T19:29:23.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.064+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 >> 172.21.15.134:0/1360155346 conn(0x7f2d680b7ea0 msgr2=0x7f2d68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:23.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.065+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 shutdown_connections 2024-03-25T19:29:23.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.065+0000 7f2d6e92e640 1 -- 172.21.15.134:0/1360155346 wait complete. 2024-03-25T19:29:23.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:23.065+0000 7f2d6e92e640 1 librados: shutdown 2024-03-25T19:29:23.078 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:29.885 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.003+0000 7f154088d640 10 monclient: get_monmap_and_config 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.003+0000 7f154088d640 10 monclient: build_initial_monmap 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.003+0000 7f154088d640 10 monclient: monmap: 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:29:30.004109+0000 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:29:30.004109+0000 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:30.003 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:30.004 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:29:30.004 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:29:30.004 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.003+0000 7f154088d640 1 Processor -- start 2024-03-25T19:29:30.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.004+0000 7f154088d640 1 -- start start 2024-03-25T19:29:30.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.004+0000 7f154088d640 10 monclient: init 2024-03-25T19:29:30.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.004+0000 7f154088d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.004+0000 7f154088d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 10 monclient(hunting): picked mon.noname-a con 0x7f153c0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 10 monclient(hunting): picked mon.noname-c con 0x7f153c13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 10 monclient(hunting): picked mon.noname-b con 0x7f153c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c142520 con 0x7f153c0a98f0 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0b28a0 con 0x7f153c13f710 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0b2a30 con 0x7f153c1430c0 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:30.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.005+0000 7f154088d640 10 monclient(hunting): authenticate will time out at 6002.163943s 2024-03-25T19:29:30.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.008+0000 7f1539d9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f153c13f710 0x7f153c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:42540/0 (socket says 172.21.15.134:42540) 2024-03-25T19:29:30.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.008+0000 7f1539d9b640 1 -- 172.21.15.134:0/645372806 learned_addr learned my addr 172.21.15.134:0/645372806 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:30.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.016+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115664341 0 0) 0x7f153c0b28a0 con 0x7f153c13f710 2024-03-25T19:29:30.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.016+0000 7f153959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:30.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.016+0000 7f153959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:30.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.016+0000 7f153959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:30.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.016+0000 7f153959a640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:29:30.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.016+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1524003520 con 0x7f153c13f710 2024-03-25T19:29:30.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.024+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182693015 0 0) 0x7f153c0b2a30 con 0x7f153c1430c0 2024-03-25T19:29:30.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.024+0000 7f153959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:30.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.024+0000 7f153959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:30.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.024+0000 7f153959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:30.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.024+0000 7f153959a640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:29:30.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.024+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153c0b28a0 con 0x7f153c1430c0 2024-03-25T19:29:30.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149992449 0 0) 0x7f1524003520 con 0x7f153c13f710 2024-03-25T19:29:30.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 >> v1:172.21.15.134:6790/0 conn(0x7f153c1430c0 legacy=0x7f153c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:30.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 >> v1:172.21.15.134:6789/0 conn(0x7f153c0a98f0 legacy=0x7f153c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:30.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 10 monclient: _finish_hunting 0 2024-03-25T19:29:30.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 1 monclient: found mon.noname-c 2024-03-25T19:29:30.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:30.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:29:30.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.027+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0b44c0 con 0x7f153c13f710 2024-03-25T19:29:30.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f153959a640 10 monclient: _finish_auth 0 2024-03-25T19:29:30.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f153959a640 10 monclient: _check_auth_tickets 2024-03-25T19:29:30.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f153959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:30.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1530001690 con 0x7f153c13f710 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f153959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f153959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f153959a640 10 monclient: dump: 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:30.030 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:30.031 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:30.031 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:30.031 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f154088d640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:29:30.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.028+0000 7f154088d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:30.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15300017f0 con 0x7f153c13f710 2024-03-25T19:29:30.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f153959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:30.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f153959a640 1 -- 172.21.15.134:0/645372806 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1530001c00 con 0x7f153c13f710 2024-03-25T19:29:30.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f153959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f153959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f153959a640 10 monclient: dump: 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:30.058 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:30.059 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f154088d640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:30.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.057+0000 7f154088d640 4 set_mon_vals no callback set 2024-03-25T19:29:30.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.058+0000 7f154088d640 10 monclient: shutdown 2024-03-25T19:29:30.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.058+0000 7f154088d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:30.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.058+0000 7f154088d640 1 -- 172.21.15.134:0/645372806 >> v1:172.21.15.159:6789/0 conn(0x7f153c13f710 legacy=0x7f153c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:30.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.058+0000 7f154088d640 1 -- 172.21.15.134:0/645372806 shutdown_connections 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.058+0000 7f154088d640 1 -- 172.21.15.134:0/645372806 >> 172.21.15.134:0/645372806 conn(0x7f153c0b7ea0 msgr2=0x7f153c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.058+0000 7f154088d640 1 -- 172.21.15.134:0/645372806 shutdown_connections 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.058+0000 7f154088d640 1 -- 172.21.15.134:0/645372806 wait complete. 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 10 monclient: build_initial_monmap 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 10 monclient: monmap: 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:30.062 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 1 librados: starting msgr at 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 1 librados: starting objecter 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 1 Processor -- start 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 1 -- start start 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 1 librados: setting wanted keys 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 1 librados: calling monclient init 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.059+0000 7f154088d640 10 monclient: init 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:30.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 10 monclient(hunting): picked mon.noname-a con 0x7f153c0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 10 monclient(hunting): picked mon.noname-c con 0x7f153c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 10 monclient(hunting): picked mon.noname-b con 0x7f153c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c138c10 con 0x7f153c0a98f0 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c138e60 con 0x7f153c1430c0 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c139140 con 0x7f153c13f710 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.060+0000 7f154088d640 10 monclient(hunting): authenticate will time out at 6002.219292s 2024-03-25T19:29:30.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.063+0000 7f153a59c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f153c0a98f0 0x7f153c137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:43206/0 (socket says 172.21.15.134:43206) 2024-03-25T19:29:30.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.063+0000 7f153a59c640 1 -- 172.21.15.134:0/2653250545 learned_addr learned my addr 172.21.15.134:0/2653250545 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:30.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.064+0000 7f152affd640 10 client.?.objecter ms_handle_connect 0x7f153c0a98f0 2024-03-25T19:29:30.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.064+0000 7f152affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:30.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.075+0000 7f152affd640 10 client.?.objecter ms_handle_connect 0x7f153c13f710 2024-03-25T19:29:30.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.075+0000 7f152affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:30.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.078+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554313009 0 0) 0x7f153c138c10 con 0x7f153c0a98f0 2024-03-25T19:29:30.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.078+0000 7f152affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:30.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.078+0000 7f152affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:30.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.078+0000 7f152affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:30.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.078+0000 7f152affd640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:29:30.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.078+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c004eb0 con 0x7f153c0a98f0 2024-03-25T19:29:30.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.080+0000 7f152affd640 10 client.?.objecter ms_handle_connect 0x7f153c1430c0 2024-03-25T19:29:30.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.080+0000 7f152affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:30.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.084+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325755017 0 0) 0x7f153c139140 con 0x7f153c13f710 2024-03-25T19:29:30.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.084+0000 7f152affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:30.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.084+0000 7f152affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:30.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.084+0000 7f152affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:30.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.084+0000 7f152affd640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:29:30.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.084+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153c138c10 con 0x7f153c13f710 2024-03-25T19:29:30.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.091+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734558108 0 0) 0x7f153c138e60 con 0x7f153c1430c0 2024-03-25T19:29:30.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.091+0000 7f152affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:30.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.091+0000 7f152affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:30.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.091+0000 7f152affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:30.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.091+0000 7f152affd640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:29:30.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.091+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153c139140 con 0x7f153c1430c0 2024-03-25T19:29:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.106+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3823943495 0 0) 0x7f153c139140 con 0x7f153c1430c0 2024-03-25T19:29:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.106+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f153c138e60 con 0x7f153c1430c0 2024-03-25T19:29:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.106+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1534001bd0 con 0x7f153c1430c0 2024-03-25T19:29:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.106+0000 7f152affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.106+0000 7f152affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.106+0000 7f152affd640 10 monclient(hunting): dump: 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.106+0000 7f152affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.110+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437835047 0 0) 0x7f151c004eb0 con 0x7f153c0a98f0 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.110+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f153c139140 con 0x7f153c0a98f0 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.110+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f152c001a30 con 0x7f153c0a98f0 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.110+0000 7f152affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.110+0000 7f152affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.110+0000 7f152affd640 10 monclient(hunting): dump: 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:30.110 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:30.111 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:30.111 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:30.111 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:30.111 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.115+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1030270068 0 0) 0x7f153c138c10 con 0x7f153c13f710 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.115+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f151c004eb0 con 0x7f153c13f710 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.115+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1530001b00 con 0x7f153c13f710 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.115+0000 7f152affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.115+0000 7f152affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.115+0000 7f152affd640 10 monclient(hunting): dump: 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:30.115 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:30.116 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 788257871 0 0) 0x7f151c004eb0 con 0x7f153c13f710 2024-03-25T19:29:30.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 >> v1:172.21.15.159:6789/0 conn(0x7f153c1430c0 legacy=0x7f153c1e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:30.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 >> v1:172.21.15.134:6789/0 conn(0x7f153c0a98f0 legacy=0x7f153c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:30.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:30.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 1 monclient: found mon.c 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c139920 con 0x7f153c13f710 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f152affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 5 monclient: authenticate success, global_id 4271 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 10 monclient: _renew_subs 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1382e0 con 0x7f153c13f710 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:30.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 10 monclient: _renew_subs 2024-03-25T19:29:30.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:30.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c0ab8b0 con 0x7f153c13f710 2024-03-25T19:29:30.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.124+0000 7f154088d640 1 librados: init done 2024-03-25T19:29:30.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.125+0000 7f154088d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:30.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.125+0000 7f154088d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:30.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.125+0000 7f154088d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:30.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.125+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c137a20 con 0x7f153c13f710 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.429+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1530001ca0 con 0x7f153c13f710 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.429+0000 7f152affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.429+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1530004c30 con 0x7f153c13f710 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.429+0000 7f152affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.429+0000 7f152affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.429+0000 7f152affd640 10 monclient: dump: 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:30.429 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:30.430 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:30.430 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:30.430 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:30.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.429+0000 7f1538d99640 4 set_mon_vals no callback set 2024-03-25T19:29:30.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.430+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f153001ac00 con 0x7f153c13f710 2024-03-25T19:29:30.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.430+0000 7f153a59c640 10 monclient: get_auth_request con 0x7f151c06e130 auth_method 0 2024-03-25T19:29:30.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.431+0000 7f152affd640 10 client.4271.objecter ms_handle_connect 0x7f151c06e130 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f15300547d0 con 0x7f153c13f710 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 10 client.4271.objecter ms_dispatch 0x7f153c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1530050110 con 0x7f153c13f710 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:30.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.435+0000 7f152affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:30.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.650+0000 7f154088d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:30.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.650+0000 7f154088d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:30.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.650+0000 7f154088d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:29:30.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.650+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f153c13d010 con 0x7f153c13f710 2024-03-25T19:29:30.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.712+0000 7f152affd640 1 -- 172.21.15.134:0/2653250545 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f1530052210 con 0x7f153c13f710 2024-03-25T19:29:30.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.712+0000 7f152affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:30.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.712+0000 7f152affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:30.712 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:30.712 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:30.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 10 librados: watch_flush enter 2024-03-25T19:29:30.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 10 librados: watch_flush exit 2024-03-25T19:29:30.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-25T19:29:30.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 10 client.4271.objecter successfully canceled tick 2024-03-25T19:29:30.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 >> v1:172.21.15.159:6800/1875874887 conn(0x7f151c06e130 legacy=0x7f151c0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 10 monclient: shutdown 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 >> v1:172.21.15.134:6790/0 conn(0x7f153c13f710 legacy=0x7f153c132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 shutdown_connections 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 >> 172.21.15.134:0/2653250545 conn(0x7f153c0b7ea0 msgr2=0x7f153c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 shutdown_connections 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 1 -- 172.21.15.134:0/2653250545 wait complete. 2024-03-25T19:29:30.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:30.715+0000 7f154088d640 1 librados: shutdown 2024-03-25T19:29:30.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:37.533 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.652+0000 7f80e8525640 10 monclient: get_monmap_and_config 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.652+0000 7f80e8525640 10 monclient: build_initial_monmap 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.652+0000 7f80e8525640 10 monclient: monmap: 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:29:37.653539+0000 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:29:37.653539+0000 2024-03-25T19:29:37.652 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:37.653 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:37.653 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:37.653 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:29:37.653 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:29:37.653 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:37.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.653+0000 7f80e8525640 1 Processor -- start 2024-03-25T19:29:37.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.653+0000 7f80e8525640 1 -- start start 2024-03-25T19:29:37.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.653+0000 7f80e8525640 10 monclient: init 2024-03-25T19:29:37.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.653+0000 7f80e8525640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:37.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.653+0000 7f80e8525640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:37.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 10 monclient(hunting): picked mon.noname-b con 0x7f80e00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:37.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 10 monclient(hunting): picked mon.noname-c con 0x7f80e013f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:37.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 10 monclient(hunting): picked mon.noname-a con 0x7f80e01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:37.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e0142520 con 0x7f80e01430c0 2024-03-25T19:29:37.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e00b28a0 con 0x7f80e013f710 2024-03-25T19:29:37.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e00b2a30 con 0x7f80e00a98f0 2024-03-25T19:29:37.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:37.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.654+0000 7f80e8525640 10 monclient(hunting): authenticate will time out at 6009.812713s 2024-03-25T19:29:37.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.661+0000 7f80e62c1640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f80e00a98f0 0x7f80e00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:40574/0 (socket says 172.21.15.134:40574) 2024-03-25T19:29:37.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.661+0000 7f80e62c1640 1 -- 172.21.15.134:0/3474754236 learned_addr learned my addr 172.21.15.134:0/3474754236 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:37.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.678+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814412025 0 0) 0x7f80e00b2a30 con 0x7f80e00a98f0 2024-03-25T19:29:37.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.678+0000 7f80e52bf640 10 monclient(hunting): _negotiate 2024-03-25T19:29:37.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.678+0000 7f80e52bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:37.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.678+0000 7f80e52bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:37.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.678+0000 7f80e52bf640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:29:37.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.678+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80cc003520 con 0x7f80e00a98f0 2024-03-25T19:29:37.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.679+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553287482 0 0) 0x7f80e0142520 con 0x7f80e01430c0 2024-03-25T19:29:37.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.679+0000 7f80e52bf640 10 monclient(hunting): _negotiate 2024-03-25T19:29:37.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.679+0000 7f80e52bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:37.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.679+0000 7f80e52bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:37.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.679+0000 7f80e52bf640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:29:37.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.679+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e00b2a30 con 0x7f80e01430c0 2024-03-25T19:29:37.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.691+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317264181 0 0) 0x7f80e00b28a0 con 0x7f80e013f710 2024-03-25T19:29:37.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.691+0000 7f80e52bf640 10 monclient(hunting): _negotiate 2024-03-25T19:29:37.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.691+0000 7f80e52bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:37.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.691+0000 7f80e52bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:37.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.691+0000 7f80e52bf640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:29:37.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.691+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e0142520 con 0x7f80e013f710 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155348739 0 0) 0x7f80e00b2a30 con 0x7f80e01430c0 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 >> v1:172.21.15.134:6790/0 conn(0x7f80e00a98f0 legacy=0x7f80e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 >> v1:172.21.15.159:6789/0 conn(0x7f80e013f710 legacy=0x7f80e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 10 monclient: _finish_hunting 0 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 1 monclient: found mon.noname-a 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e00b44c0 con 0x7f80e01430c0 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.692+0000 7f80e52bf640 10 monclient: _finish_auth 0 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e52bf640 10 monclient: _check_auth_tickets 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e52bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:37.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80d80019f0 con 0x7f80e01430c0 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e52bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e52bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e52bf640 10 monclient: dump: 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e8525640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:29:37.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.693+0000 7f80e8525640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80d8001b50 con 0x7f80e01430c0 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e52bf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e52bf640 1 -- 172.21.15.134:0/3474754236 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80d8001f60 con 0x7f80e01430c0 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e8525640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e8525640 4 set_mon_vals no callback set 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e52bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e52bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.697+0000 7f80e52bf640 10 monclient: dump: 2024-03-25T19:29:37.697 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:37.698 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:37.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.698+0000 7f80e8525640 10 monclient: shutdown 2024-03-25T19:29:37.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.698+0000 7f80e8525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:37.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.698+0000 7f80e8525640 1 -- 172.21.15.134:0/3474754236 >> v1:172.21.15.134:6789/0 conn(0x7f80e01430c0 legacy=0x7f80e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:37.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.698+0000 7f80e8525640 1 -- 172.21.15.134:0/3474754236 shutdown_connections 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.698+0000 7f80e8525640 1 -- 172.21.15.134:0/3474754236 >> 172.21.15.134:0/3474754236 conn(0x7f80e00b7ea0 msgr2=0x7f80e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.698+0000 7f80e8525640 1 -- 172.21.15.134:0/3474754236 shutdown_connections 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.698+0000 7f80e8525640 1 -- 172.21.15.134:0/3474754236 wait complete. 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 10 monclient: build_initial_monmap 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 10 monclient: monmap: 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:29:37.701 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 1 librados: starting msgr at 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 1 librados: starting objecter 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 1 Processor -- start 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 1 -- start start 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 1 librados: setting wanted keys 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 1 librados: calling monclient init 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.699+0000 7f80e8525640 10 monclient: init 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 10 monclient(hunting): picked mon.noname-b con 0x7f80e00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 10 monclient(hunting): picked mon.noname-a con 0x7f80e013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 10 monclient(hunting): picked mon.noname-c con 0x7f80e01430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e0138bd0 con 0x7f80e013f710 2024-03-25T19:29:37.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e00aba30 con 0x7f80e00a98f0 2024-03-25T19:29:37.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e0137740 con 0x7f80e01430c0 2024-03-25T19:29:37.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:37.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.700+0000 7f80e8525640 10 monclient(hunting): authenticate will time out at 6009.859429s 2024-03-25T19:29:37.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.713+0000 7f80e5ac0640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f80e013f710 0x7f80e0134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:38460/0 (socket says 172.21.15.134:38460) 2024-03-25T19:29:37.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.713+0000 7f80e5ac0640 1 -- 172.21.15.134:0/1663270246 learned_addr learned my addr 172.21.15.134:0/1663270246 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:37.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.715+0000 7f80d6ffd640 10 client.?.objecter ms_handle_connect 0x7f80e013f710 2024-03-25T19:29:37.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.715+0000 7f80d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:37.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.721+0000 7f80d6ffd640 10 client.?.objecter ms_handle_connect 0x7f80e00a98f0 2024-03-25T19:29:37.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.721+0000 7f80d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:37.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.722+0000 7f80d6ffd640 10 client.?.objecter ms_handle_connect 0x7f80e01430c0 2024-03-25T19:29:37.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.722+0000 7f80d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:37.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.725+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260127302 0 0) 0x7f80e0138bd0 con 0x7f80e013f710 2024-03-25T19:29:37.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.725+0000 7f80d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:37.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.725+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:37.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.725+0000 7f80d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:37.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.725+0000 7f80d6ffd640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:29:37.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.725+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c4004eb0 con 0x7f80e013f710 2024-03-25T19:29:37.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.735+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753112341 0 0) 0x7f80e00aba30 con 0x7f80e00a98f0 2024-03-25T19:29:37.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.735+0000 7f80d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:37.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.735+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:37.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.735+0000 7f80d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:37.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.735+0000 7f80d6ffd640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:29:37.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.735+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e0138bd0 con 0x7f80e00a98f0 2024-03-25T19:29:37.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.738+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719169630 0 0) 0x7f80e0137740 con 0x7f80e01430c0 2024-03-25T19:29:37.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.738+0000 7f80d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:37.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.738+0000 7f80d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:37.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.738+0000 7f80d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:37.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.738+0000 7f80d6ffd640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:29:37.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.738+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e00aba30 con 0x7f80e01430c0 2024-03-25T19:29:37.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.748+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 344127034 0 0) 0x7f80c4004eb0 con 0x7f80e013f710 2024-03-25T19:29:37.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.748+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80e0137740 con 0x7f80e013f710 2024-03-25T19:29:37.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.748+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80d0001a10 con 0x7f80e013f710 2024-03-25T19:29:37.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.748+0000 7f80d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:37.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.748+0000 7f80d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:37.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.748+0000 7f80d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:37.749 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:37.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.748+0000 7f80d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:37.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.759+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 142017598 0 0) 0x7f80e0138bd0 con 0x7f80e00a98f0 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.759+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80c4004eb0 con 0x7f80e00a98f0 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.759+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80dc001950 con 0x7f80e00a98f0 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.759+0000 7f80d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.759+0000 7f80d6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.759+0000 7f80d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:37.760 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.760+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906473490 0 0) 0x7f80e00aba30 con 0x7f80e01430c0 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.760+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80e0138bd0 con 0x7f80e01430c0 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.760+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80d8001e60 con 0x7f80e01430c0 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.760+0000 7f80d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.760+0000 7f80d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.760+0000 7f80d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:37.761 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:37.762 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:37.762 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.764+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1649507746 0 0) 0x7f80e0137740 con 0x7f80e013f710 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.764+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 >> v1:172.21.15.134:6790/0 conn(0x7f80e01430c0 legacy=0x7f80e01d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.764+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 >> v1:172.21.15.159:6789/0 conn(0x7f80e00a98f0 legacy=0x7f80e01384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.764+0000 7f80d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.764+0000 7f80d6ffd640 1 monclient: found mon.a 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80d6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0134de0 con 0x7f80e013f710 2024-03-25T19:29:37.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 5 monclient: authenticate success, global_id 4293 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 10 monclient: _renew_subs 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e00b1620 con 0x7f80e013f710 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 10 monclient: _renew_subs 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e0134900 con 0x7f80e013f710 2024-03-25T19:29:37.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.765+0000 7f80e8525640 1 librados: init done 2024-03-25T19:29:37.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.766+0000 7f80e8525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:37.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.766+0000 7f80e8525640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:37.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.766+0000 7f80e8525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:37.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:37.766+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e00627b0 con 0x7f80e013f710 2024-03-25T19:29:38.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.061+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80d0001c70 con 0x7f80e013f710 2024-03-25T19:29:38.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.061+0000 7f80d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:38.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.061+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80d0003a60 con 0x7f80e013f710 2024-03-25T19:29:38.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.061+0000 7f80d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:38.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.061+0000 7f80d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.061+0000 7f80d6ffd640 10 monclient: dump: 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:38.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.061+0000 7f80e4abe640 4 set_mon_vals no callback set 2024-03-25T19:29:38.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.062+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f80d0003c80 con 0x7f80e013f710 2024-03-25T19:29:38.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.063+0000 7f80e62c1640 10 monclient: get_auth_request con 0x7f80c406e0f0 auth_method 0 2024-03-25T19:29:38.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.063+0000 7f80d6ffd640 10 client.4293.objecter ms_handle_connect 0x7f80c406e0f0 2024-03-25T19:29:38.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f80d00548b0 con 0x7f80e013f710 2024-03-25T19:29:38.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 10 client.4293.objecter ms_dispatch 0x7f80e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:38.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:38.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:38.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T19:29:38.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80d00503c0 con 0x7f80e013f710 2024-03-25T19:29:38.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:38.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.068+0000 7f80d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:38.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.327+0000 7f80e8525640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:38.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.327+0000 7f80e8525640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:38.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.327+0000 7f80e8525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:38.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.327+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80e01d7360 con 0x7f80e013f710 2024-03-25T19:29:38.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.351+0000 7f80d6ffd640 1 -- 172.21.15.134:0/1663270246 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f80d00522f0 con 0x7f80e013f710 2024-03-25T19:29:38.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.351+0000 7f80d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:38.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.351+0000 7f80d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:38.351 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:38.351 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 10 librados: watch_flush enter 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 10 librados: watch_flush exit 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 10 client.4293.objecter successfully canceled tick 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 >> v1:172.21.15.159:6800/1875874887 conn(0x7f80c406e0f0 legacy=0x7f80c40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 10 monclient: shutdown 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.353+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 >> v1:172.21.15.134:6789/0 conn(0x7f80e013f710 legacy=0x7f80e0134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.354+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 shutdown_connections 2024-03-25T19:29:38.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.354+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 >> 172.21.15.134:0/1663270246 conn(0x7f80e00b7ea0 msgr2=0x7f80e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:38.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.354+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 shutdown_connections 2024-03-25T19:29:38.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.354+0000 7f80e8525640 1 -- 172.21.15.134:0/1663270246 wait complete. 2024-03-25T19:29:38.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:38.354+0000 7f80e8525640 1 librados: shutdown 2024-03-25T19:29:38.364 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:39.594 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:44.266 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.649 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:44.321 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.700 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:44.372 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.714 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:44.361 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.723 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:44.395 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.730 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:44.377 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.735 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:44.407 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.780 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:44.427 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:39.805 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:44.478 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:45.171 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.290+0000 7f9acdcc1640 10 monclient: get_monmap_and_config 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.290+0000 7f9acdcc1640 10 monclient: build_initial_monmap 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.290+0000 7f9acdcc1640 10 monclient: monmap: 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:29:45.291277+0000 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:29:45.291277+0000 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:45.290 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:45.291 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:29:45.291 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:29:45.291 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.291+0000 7f9acdcc1640 1 Processor -- start 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.291+0000 7f9acdcc1640 1 -- start start 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.291+0000 7f9acdcc1640 10 monclient: init 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.291+0000 7f9acdcc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac813fc60 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac8070570 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac80727a0 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac813c9f0 con 0x7f9ac8070570 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac806fac0 con 0x7f9ac80727a0 2024-03-25T19:29:45.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac80715d0 con 0x7f9ac813fc60 2024-03-25T19:29:45.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:45.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.292+0000 7f9acdcc1640 10 monclient(hunting): authenticate will time out at 6017.450751s 2024-03-25T19:29:45.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.297+0000 7f9abffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9ac8070570 0x7f9ac8072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:37304/0 (socket says 172.21.15.134:37304) 2024-03-25T19:29:45.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.297+0000 7f9abffff640 1 -- 172.21.15.134:0/1699916633 learned_addr learned my addr 172.21.15.134:0/1699916633 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:45.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.311+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161642903 0 0) 0x7f9ac813c9f0 con 0x7f9ac8070570 2024-03-25T19:29:45.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.311+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:45.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.311+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:45.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.311+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:45.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.311+0000 7f9ac6ffd640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:29:45.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.311+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab4003520 con 0x7f9ac8070570 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.320+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431345843 0 0) 0x7f9ab4003520 con 0x7f9ac8070570 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.320+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6790/0 conn(0x7f9ac813fc60 legacy=0x7f9ac806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.320+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.159:6789/0 conn(0x7f9ac80727a0 legacy=0x7f9ac814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.320+0000 7f9ac6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.320+0000 7f9ac6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac80718b0 con 0x7f9ac8070570 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:45.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9ab0001690 con 0x7f9ac8070570 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9ac6ffd640 10 monclient: dump: 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:45.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9acdcc1640 5 monclient: authenticate success, global_id 4296 2024-03-25T19:29:45.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.321+0000 7f9acdcc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9abffff640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6789/0 conn(0x7f9ac8070570 legacy=0x7f9ac8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9abffff640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6789/0 conn(0x7f9ac8070570 legacy=0x7f9ac8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9abffff640 1 --1- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6789/0 conn(0x7f9ac8070570 0x7f9ac8072090 :-1 s=OPENED pgs=5458 cs=1 l=1).handle_message read tag failed 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9abffff640 1 --1- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6789/0 conn(0x7f9ac8070570 0x7f9ac8072090 :-1 s=OPENED pgs=5458 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.134:6789/0 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6789/0 conn(0x7f9ac8070570 legacy=0x7f9ac8072090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient(hunting): picked mon.a con 0x7f9ab4003af0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient(hunting): picked mon.b con 0x7f9ac80727a0 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:45.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient(hunting): picked mon.c con 0x7f9ac813fc60 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:45.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9ab4003520 con 0x7f9ab4003af0 2024-03-25T19:29:45.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9ac813c9f0 con 0x7f9ac80727a0 2024-03-25T19:29:45.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9ab4010e40 con 0x7f9ac813fc60 2024-03-25T19:29:45.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:45.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.322+0000 7f9ac6ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.134:6789/0 2024-03-25T19:29:45.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.395+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442390727 0 0) 0x7f9ab4003520 con 0x7f9ab4003af0 2024-03-25T19:29:45.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.395+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:45.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.395+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:45.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.395+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:29:45.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.395+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.134:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f9ab4010fd0 con 0x7f9ab4003af0 2024-03-25T19:29:45.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.404+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786484574 0 0) 0x7f9ac813c9f0 con 0x7f9ac80727a0 2024-03-25T19:29:45.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.404+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:45.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.404+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:45.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.404+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:29:45.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.404+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.159:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f9ab4003520 con 0x7f9ac80727a0 2024-03-25T19:29:45.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.413+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252129305 0 0) 0x7f9ab4010e40 con 0x7f9ac813fc60 2024-03-25T19:29:45.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.413+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:45.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.413+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:45.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.413+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:29:45.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.413+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.134:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f9ac813c9f0 con 0x7f9ac813fc60 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 714959037 0 0) 0x7f9ab4010fd0 con 0x7f9ab4003af0 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6790/0 conn(0x7f9ac813fc60 legacy=0x7f9ab400d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.159:6789/0 conn(0x7f9ac80727a0 legacy=0x7f9ab4009aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 1 monclient: found mon.a 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f9ab4012160 con 0x7f9ab4003af0 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:45.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9ab8001660 con 0x7f9ab4003af0 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9ac6ffd640 10 monclient: dump: 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:45.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.419+0000 7f9acdcc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:45.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.425+0000 7f9ac6ffd640 1 -- 172.21.15.134:0/1699916633 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ab80017c0 con 0x7f9ab4003af0 2024-03-25T19:29:45.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.425+0000 7f9ac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:45.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.425+0000 7f9acdcc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:45.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.425+0000 7f9acdcc1640 4 set_mon_vals no callback set 2024-03-25T19:29:45.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.426+0000 7f9acdcc1640 10 monclient: shutdown 2024-03-25T19:29:45.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.426+0000 7f9acdcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:45.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.426+0000 7f9acdcc1640 1 -- 172.21.15.134:0/1699916633 >> v1:172.21.15.134:6789/0 conn(0x7f9ab4003af0 legacy=0x7f9ab4003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.426+0000 7f9acdcc1640 1 -- 172.21.15.134:0/1699916633 shutdown_connections 2024-03-25T19:29:45.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.426+0000 7f9acdcc1640 1 -- 172.21.15.134:0/1699916633 >> 172.21.15.134:0/1699916633 conn(0x7f9ac80b4a80 msgr2=0x7f9ac80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:45.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.426+0000 7f9acdcc1640 1 -- 172.21.15.134:0/1699916633 shutdown_connections 2024-03-25T19:29:45.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.426+0000 7f9acdcc1640 1 -- 172.21.15.134:0/1699916633 wait complete. 2024-03-25T19:29:45.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 10 monclient: build_initial_monmap 2024-03-25T19:29:45.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 10 monclient: monmap: 2024-03-25T19:29:45.427 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:45.427 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 1 librados: starting msgr at 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 1 librados: starting objecter 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 1 Processor -- start 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 1 -- start start 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 1 librados: setting wanted keys 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 1 librados: calling monclient init 2024-03-25T19:29:45.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.427+0000 7f9acdcc1640 10 monclient: init 2024-03-25T19:29:45.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:45.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab4003af0 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac80727a0 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac813fc60 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac813f1e0 con 0x7f9ac80727a0 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac80ac6f0 con 0x7f9ab4003af0 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac80aab90 con 0x7f9ac813fc60 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:45.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.428+0000 7f9acdcc1640 10 monclient(hunting): authenticate will time out at 6017.587157s 2024-03-25T19:29:45.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.434+0000 7f9ac77fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9ab4003af0 0x7f9ac81272d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52440/0 (socket says 172.21.15.134:52440) 2024-03-25T19:29:45.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.434+0000 7f9ac77fe640 1 -- 172.21.15.134:0/4017947302 learned_addr learned my addr 172.21.15.134:0/4017947302 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:45.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.436+0000 7f9ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f9ab4003af0 2024-03-25T19:29:45.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.436+0000 7f9ac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:45.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.440+0000 7f9ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f9ac80727a0 2024-03-25T19:29:45.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.440+0000 7f9ac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:45.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f9ac813fc60 2024-03-25T19:29:45.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:45.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121725496 0 0) 0x7f9ac813f1e0 con 0x7f9ac80727a0 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.447+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8004eb0 con 0x7f9ac80727a0 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.448+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528146963 0 0) 0x7f9ac80ac6f0 con 0x7f9ab4003af0 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.448+0000 7f9ac4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.448+0000 7f9ac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.448+0000 7f9ac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.448+0000 7f9ac4ff9640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:29:45.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.448+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac813f1e0 con 0x7f9ab4003af0 2024-03-25T19:29:45.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.458+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481233705 0 0) 0x7f9ac813f1e0 con 0x7f9ab4003af0 2024-03-25T19:29:45.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.458+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ac80ac6f0 con 0x7f9ab4003af0 2024-03-25T19:29:45.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.458+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9ab8001b50 con 0x7f9ab4003af0 2024-03-25T19:29:45.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.458+0000 7f9ac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.458+0000 7f9ac4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.458+0000 7f9ac4ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:45.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.458+0000 7f9ac4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:45.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.460+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751537872 0 0) 0x7f9ac80aab90 con 0x7f9ac813fc60 2024-03-25T19:29:45.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.460+0000 7f9ac4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:45.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.460+0000 7f9ac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:45.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.460+0000 7f9ac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:45.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.460+0000 7f9ac4ff9640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:29:45.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.460+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac813f1e0 con 0x7f9ac813fc60 2024-03-25T19:29:45.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.470+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4079676539 0 0) 0x7f9aa8004eb0 con 0x7f9ac80727a0 2024-03-25T19:29:45.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.470+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ac80aab90 con 0x7f9ac80727a0 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.470+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9ab0001850 con 0x7f9ac80727a0 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.470+0000 7f9ac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.470+0000 7f9ac4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.470+0000 7f9ac4ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:45.471 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:45.472 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1767279988 0 0) 0x7f9ac80ac6f0 con 0x7f9ab4003af0 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 >> v1:172.21.15.134:6790/0 conn(0x7f9ac813fc60 legacy=0x7f9ac81e1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 >> v1:172.21.15.134:6789/0 conn(0x7f9ac80727a0 legacy=0x7f9ac80aa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 1 monclient: found mon.b 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:45.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac80adad0 con 0x7f9ab4003af0 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9ac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 10 monclient: _renew_subs 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac80addb0 con 0x7f9ab4003af0 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 10 monclient: _renew_subs 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac80acbf0 con 0x7f9ab4003af0 2024-03-25T19:29:45.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.481+0000 7f9acdcc1640 1 librados: init done 2024-03-25T19:29:45.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.482+0000 7f9acdcc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:45.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.482+0000 7f9acdcc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:45.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.482+0000 7f9acdcc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:45.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.482+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac80ac910 con 0x7f9ab4003af0 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.771+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ab8001d50 con 0x7f9ab4003af0 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.771+0000 7f9ac4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.771+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9ab8003a40 con 0x7f9ab4003af0 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.771+0000 7f9ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.771+0000 7f9ac4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.771+0000 7f9ac4ff9640 10 monclient: dump: 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:45.771 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.772 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:45.772 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:45.772 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:45.772 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:45.772 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:45.772 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:45.772 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:45.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.771+0000 7f9ac67fc640 4 set_mon_vals no callback set 2024-03-25T19:29:45.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.772+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f9ab8003be0 con 0x7f9ab4003af0 2024-03-25T19:29:45.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.772+0000 7f9abffff640 10 monclient: get_auth_request con 0x7f9aa806df70 auth_method 0 2024-03-25T19:29:45.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.772+0000 7f9ac4ff9640 10 client.4282.objecter ms_handle_connect 0x7f9aa806df70 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f9ab8054950 con 0x7f9ab4003af0 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 10 client.4282.objecter ms_dispatch 0x7f9ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ab8050290 con 0x7f9ab4003af0 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:45.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:45.777+0000 7f9ac4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:46.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.033+0000 7f9acdcc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:46.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.033+0000 7f9acdcc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:46.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.033+0000 7f9acdcc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:46.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.033+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ac80627b0 con 0x7f9ab4003af0 2024-03-25T19:29:46.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.067+0000 7f9ac4ff9640 1 -- 172.21.15.134:0/4017947302 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f9ab8052390 con 0x7f9ab4003af0 2024-03-25T19:29:46.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.067+0000 7f9ac4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:46.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.067+0000 7f9ac4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:46.068 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:46.068 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:46.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 10 librados: watch_flush enter 2024-03-25T19:29:46.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 10 librados: watch_flush exit 2024-03-25T19:29:46.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 >> v1:172.21.15.159:6800/1875874887 conn(0x7f9aa806df70 legacy=0x7f9aa8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 10 monclient: shutdown 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 >> v1:172.21.15.159:6789/0 conn(0x7f9ab4003af0 legacy=0x7f9ac81272d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 shutdown_connections 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 >> 172.21.15.134:0/4017947302 conn(0x7f9ac80b4a80 msgr2=0x7f9ac806f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 shutdown_connections 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 1 -- 172.21.15.134:0/4017947302 wait complete. 2024-03-25T19:29:46.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:46.070+0000 7f9acdcc1640 1 librados: shutdown 2024-03-25T19:29:46.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:52.890 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.006+0000 7ff564e33640 10 monclient: get_monmap_and_config 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.006+0000 7ff564e33640 10 monclient: build_initial_monmap 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.006+0000 7ff564e33640 10 monclient: monmap: 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:29:53.007440+0000 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:29:53.007440+0000 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:53.006 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:53.007 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:29:53.007 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:29:53.007 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 1 Processor -- start 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 1 -- start start 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient: init 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient(hunting): picked mon.noname-a con 0x7ff5600b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient(hunting): picked mon.noname-c con 0x7ff5600b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient(hunting): picked mon.noname-b con 0x7ff56014be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600af940 con 0x7ff5600b4d60 2024-03-25T19:29:53.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600b0a70 con 0x7ff5600b1730 2024-03-25T19:29:53.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600b0d50 con 0x7ff56014be90 2024-03-25T19:29:53.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:53.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.007+0000 7ff564e33640 10 monclient(hunting): authenticate will time out at 6025.166541s 2024-03-25T19:29:53.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.020+0000 7ff557fff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff5600b1730 0x7ff5600b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:36408/0 (socket says 172.21.15.134:36408) 2024-03-25T19:29:53.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.020+0000 7ff557fff640 1 -- 172.21.15.134:0/2354716467 learned_addr learned my addr 172.21.15.134:0/2354716467 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:53.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.041+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229945475 0 0) 0x7ff5600af940 con 0x7ff5600b4d60 2024-03-25T19:29:53.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.041+0000 7ff55dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:53.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.041+0000 7ff55dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:53.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.041+0000 7ff55dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:53.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.041+0000 7ff55dd9b640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:29:53.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.041+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff54c003520 con 0x7ff5600b4d60 2024-03-25T19:29:53.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.043+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912401462 0 0) 0x7ff5600b0a70 con 0x7ff5600b1730 2024-03-25T19:29:53.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.043+0000 7ff55dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:53.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.043+0000 7ff55dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:53.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.043+0000 7ff55dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.043+0000 7ff55dd9b640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.043+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5600af940 con 0x7ff5600b1730 2024-03-25T19:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.047+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981988267 0 0) 0x7ff5600b0d50 con 0x7ff56014be90 2024-03-25T19:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.047+0000 7ff55dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.047+0000 7ff55dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.047+0000 7ff55dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.047+0000 7ff55dd9b640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.047+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5600b0a70 con 0x7ff56014be90 2024-03-25T19:29:53.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326460966 0 0) 0x7ff5600b0a70 con 0x7ff56014be90 2024-03-25T19:29:53.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 >> v1:172.21.15.159:6789/0 conn(0x7ff5600b1730 legacy=0x7ff5600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:53.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 >> v1:172.21.15.134:6789/0 conn(0x7ff5600b4d60 legacy=0x7ff56013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:53.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:29:53.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 1 monclient: found mon.noname-b 2024-03-25T19:29:53.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600b1030 con 0x7ff56014be90 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff558001930 con 0x7ff56014be90 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff564e33640 5 monclient: authenticate success, global_id 4283 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff55dd9b640 10 monclient: dump: 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:53.061 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:53.062 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:53.062 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:53.062 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:53.062 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:53.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.060+0000 7ff564e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff558001a90 con 0x7ff56014be90 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff55dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff55dd9b640 1 -- 172.21.15.134:0/2354716467 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff558001ee0 con 0x7ff56014be90 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff564e33640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff564e33640 4 set_mon_vals no callback set 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff55dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff55dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.098+0000 7ff55dd9b640 10 monclient: dump: 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.098 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:53.099 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:53.099 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:53.099 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:53.099 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:53.099 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:53.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.099+0000 7ff564e33640 10 monclient: shutdown 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.099+0000 7ff564e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.099+0000 7ff564e33640 1 -- 172.21.15.134:0/2354716467 >> v1:172.21.15.134:6790/0 conn(0x7ff56014be90 legacy=0x7ff56014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.099+0000 7ff564e33640 1 -- 172.21.15.134:0/2354716467 shutdown_connections 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.099+0000 7ff564e33640 1 -- 172.21.15.134:0/2354716467 >> 172.21.15.134:0/2354716467 conn(0x7ff5600617a0 msgr2=0x7ff560061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.099+0000 7ff564e33640 1 -- 172.21.15.134:0/2354716467 shutdown_connections 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.099+0000 7ff564e33640 1 -- 172.21.15.134:0/2354716467 wait complete. 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.100+0000 7ff564e33640 10 monclient: build_initial_monmap 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.100+0000 7ff564e33640 10 monclient: monmap: 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:29:53.101 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.100+0000 7ff564e33640 1 librados: starting msgr at 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.100+0000 7ff564e33640 1 librados: starting objecter 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 1 Processor -- start 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 1 -- start start 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 1 librados: setting wanted keys 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 1 librados: calling monclient init 2024-03-25T19:29:53.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 10 monclient: init 2024-03-25T19:29:53.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:53.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:53.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.101+0000 7ff564e33640 10 monclient(hunting): picked mon.noname-b con 0x7ff5600b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:29:53.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.102+0000 7ff564e33640 10 monclient(hunting): picked mon.noname-c con 0x7ff5600b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:29:53.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.102+0000 7ff564e33640 10 monclient(hunting): picked mon.noname-a con 0x7ff56014be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:29:53.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.102+0000 7ff564e33640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5601daea0 con 0x7ff56014be90 2024-03-25T19:29:53.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.102+0000 7ff564e33640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56013b690 con 0x7ff5600b1730 2024-03-25T19:29:53.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.102+0000 7ff564e33640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56013a3e0 con 0x7ff5600b4d60 2024-03-25T19:29:53.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.102+0000 7ff564e33640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:53.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.102+0000 7ff564e33640 10 monclient(hunting): authenticate will time out at 6025.260918s 2024-03-25T19:29:53.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.105+0000 7ff55e59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff5600b1730 0x7ff560139cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:36462/0 (socket says 172.21.15.134:36462) 2024-03-25T19:29:53.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.105+0000 7ff55e59c640 1 -- 172.21.15.134:0/2883666469 learned_addr learned my addr 172.21.15.134:0/2883666469 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:29:53.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.106+0000 7ff556ffd640 10 client.?.objecter ms_handle_connect 0x7ff5600b1730 2024-03-25T19:29:53.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.106+0000 7ff556ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:53.111 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.111+0000 7ff556ffd640 10 client.?.objecter ms_handle_connect 0x7ff56014be90 2024-03-25T19:29:53.111 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.111+0000 7ff556ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:53.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.116+0000 7ff556ffd640 10 client.?.objecter ms_handle_connect 0x7ff5600b4d60 2024-03-25T19:29:53.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.116+0000 7ff556ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:53.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.117+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414648058 0 0) 0x7ff56013b690 con 0x7ff5600b1730 2024-03-25T19:29:53.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.117+0000 7ff556ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:53.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.117+0000 7ff556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:53.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.117+0000 7ff556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:53.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.117+0000 7ff556ffd640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:29:53.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.117+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540004eb0 con 0x7ff5600b1730 2024-03-25T19:29:53.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.124+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 85031834 0 0) 0x7ff540004eb0 con 0x7ff5600b1730 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.124+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff56013b690 con 0x7ff5600b1730 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.124+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff5500019b0 con 0x7ff5600b1730 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.124+0000 7ff556ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.124+0000 7ff556ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.124+0000 7ff556ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:53.125 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:53.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.124+0000 7ff556ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:53.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.129+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408736164 0 0) 0x7ff5601daea0 con 0x7ff56014be90 2024-03-25T19:29:53.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.129+0000 7ff556ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:53.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.129+0000 7ff556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:53.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.129+0000 7ff556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:53.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.129+0000 7ff556ffd640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:29:53.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.130+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540004eb0 con 0x7ff56014be90 2024-03-25T19:29:53.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.130+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192864233 0 0) 0x7ff56013a3e0 con 0x7ff5600b4d60 2024-03-25T19:29:53.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.130+0000 7ff556ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:53.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.130+0000 7ff556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:53.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.130+0000 7ff556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:53.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.130+0000 7ff556ffd640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:29:53.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.130+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5601daea0 con 0x7ff5600b4d60 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 589400184 0 0) 0x7ff56013b690 con 0x7ff5600b1730 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 >> v1:172.21.15.134:6790/0 conn(0x7ff5600b4d60 legacy=0x7ff5601db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 >> v1:172.21.15.134:6789/0 conn(0x7ff56014be90 legacy=0x7ff560132e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 1 monclient: found mon.b 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5601dba90 con 0x7ff5600b1730 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:53.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 10 monclient: _renew_subs 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56013d0f0 con 0x7ff5600b1730 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 10 monclient: _renew_subs 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff560139210 con 0x7ff5600b1730 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.131+0000 7ff564e33640 1 librados: init done 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.132+0000 7ff564e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.132+0000 7ff564e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.132+0000 7ff564e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:53.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.132+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5601dad00 con 0x7ff5600b1730 2024-03-25T19:29:53.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.413+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff550001d10 con 0x7ff5600b1730 2024-03-25T19:29:53.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.413+0000 7ff556ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:53.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.413+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff5500038c0 con 0x7ff5600b1730 2024-03-25T19:29:53.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.413+0000 7ff556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:53.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.413+0000 7ff556ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.413+0000 7ff556ffd640 10 monclient: dump: 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:29:53.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.413+0000 7ff55d59a640 4 set_mon_vals no callback set 2024-03-25T19:29:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.414+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7ff550003ae0 con 0x7ff5600b1730 2024-03-25T19:29:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.414+0000 7ff557fff640 10 monclient: get_auth_request con 0x7ff54006de80 auth_method 0 2024-03-25T19:29:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.415+0000 7ff556ffd640 10 client.4288.objecter ms_handle_connect 0x7ff54006de80 2024-03-25T19:29:53.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.420+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7ff550054690 con 0x7ff5600b1730 2024-03-25T19:29:53.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.420+0000 7ff556ffd640 10 client.4288.objecter ms_dispatch 0x7ff560000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:53.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.420+0000 7ff556ffd640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:53.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.420+0000 7ff556ffd640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:53.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.421+0000 7ff556ffd640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:29:53.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.422+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5500501a0 con 0x7ff5600b1730 2024-03-25T19:29:53.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.422+0000 7ff556ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:53.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.422+0000 7ff556ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:53.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.649+0000 7ff564e33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:53.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.649+0000 7ff564e33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:53.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.649+0000 7ff564e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:29:53.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.649+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5600627b0 con 0x7ff5600b1730 2024-03-25T19:29:53.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.674+0000 7ff556ffd640 1 -- 172.21.15.134:0/2883666469 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7ff5500520d0 con 0x7ff5600b1730 2024-03-25T19:29:53.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.674+0000 7ff556ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:53.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.674+0000 7ff556ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:53.675 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:29:53.675 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:53.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 10 librados: watch_flush enter 2024-03-25T19:29:53.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 10 librados: watch_flush exit 2024-03-25T19:29:53.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:29:53.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:29:53.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 >> v1:172.21.15.159:6800/1875874887 conn(0x7ff54006de80 legacy=0x7ff540070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:53.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 10 monclient: shutdown 2024-03-25T19:29:53.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:53.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 >> v1:172.21.15.159:6789/0 conn(0x7ff5600b1730 legacy=0x7ff560139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:53.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 shutdown_connections 2024-03-25T19:29:53.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 >> 172.21.15.134:0/2883666469 conn(0x7ff5600617a0 msgr2=0x7ff5600b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:53.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 shutdown_connections 2024-03-25T19:29:53.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 1 -- 172.21.15.134:0/2883666469 wait complete. 2024-03-25T19:29:53.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:29:53.677+0000 7ff564e33640 1 librados: shutdown 2024-03-25T19:29:53.688 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:59.673 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:04.345 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.686 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:04.359 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.718 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:04.390 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.730 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:04.377 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.735 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:04.407 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.736 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:04.383 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.742 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:04.414 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.769 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:04.416 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:59.784 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:04.456 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:00.494 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:00.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.607+0000 7fc23a0d9640 10 monclient: get_monmap_and_config 2024-03-25T19:30:00.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.607+0000 7fc23a0d9640 10 monclient: build_initial_monmap 2024-03-25T19:30:00.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.607+0000 7fc23a0d9640 10 monclient: monmap: 2024-03-25T19:30:00.607 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:00.608756+0000 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:00.608756+0000 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:00.608 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:00.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.608+0000 7fc23a0d9640 1 Processor -- start 2024-03-25T19:30:00.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.608+0000 7fc23a0d9640 1 -- start start 2024-03-25T19:30:00.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.608+0000 7fc23a0d9640 10 monclient: init 2024-03-25T19:30:00.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:00.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc2340a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc23413f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc2341430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234142520 con 0x7fc23413f710 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340b28a0 con 0x7fc2340a98f0 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340b2a30 con 0x7fc2341430c0 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:00.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.609+0000 7fc23a0d9640 10 monclient(hunting): authenticate will time out at 6032.768012s 2024-03-25T19:30:00.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.627+0000 7fc2337fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc2340a98f0 0x7fc2340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:40992/0 (socket says 172.21.15.134:40992) 2024-03-25T19:30:00.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.627+0000 7fc2337fe640 1 -- 172.21.15.134:0/1117601866 learned_addr learned my addr 172.21.15.134:0/1117601866 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:00.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.645+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151118149 0 0) 0x7fc2340b2a30 con 0x7fc2341430c0 2024-03-25T19:30:00.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.645+0000 7fc2327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.645+0000 7fc2327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.645+0000 7fc2327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.645+0000 7fc2327fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:30:00.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.645+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21c003520 con 0x7fc2341430c0 2024-03-25T19:30:00.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.648+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761049205 0 0) 0x7fc2340b28a0 con 0x7fc2340a98f0 2024-03-25T19:30:00.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.648+0000 7fc2327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.648+0000 7fc2327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.648+0000 7fc2327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.648+0000 7fc2327fc640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:30:00.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.648+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2340b2a30 con 0x7fc2340a98f0 2024-03-25T19:30:00.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.658+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749463515 0 0) 0x7fc234142520 con 0x7fc23413f710 2024-03-25T19:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.658+0000 7fc2327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.658+0000 7fc2327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.658+0000 7fc2327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.658+0000 7fc2327fc640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.658+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2340b28a0 con 0x7fc23413f710 2024-03-25T19:30:00.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3491353433 0 0) 0x7fc2340b28a0 con 0x7fc23413f710 2024-03-25T19:30:00.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 >> v1:172.21.15.134:6790/0 conn(0x7fc2341430c0 legacy=0x7fc23414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 >> v1:172.21.15.159:6789/0 conn(0x7fc2340a98f0 legacy=0x7fc2340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:00.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 1 monclient: found mon.noname-a 2024-03-25T19:30:00.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340b44c0 con 0x7fc23413f710 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc2280015b0 con 0x7fc23413f710 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc2327fc640 10 monclient: dump: 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.686 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.687 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:00.687 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:00.687 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:00.687 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:00.687 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:00.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc23a0d9640 5 monclient: authenticate success, global_id 4308 2024-03-25T19:30:00.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.685+0000 7fc23a0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:00.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc228001710 con 0x7fc23413f710 2024-03-25T19:30:00.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc2327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:00.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc2327fc640 1 -- 172.21.15.134:0/1117601866 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc228001b60 con 0x7fc23413f710 2024-03-25T19:30:00.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc2327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc2327fc640 10 monclient: dump: 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc23a0d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:00.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.707+0000 7fc23a0d9640 4 set_mon_vals no callback set 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.708+0000 7fc23a0d9640 10 monclient: shutdown 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.708+0000 7fc23a0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.708+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1117601866 >> v1:172.21.15.134:6789/0 conn(0x7fc23413f710 legacy=0x7fc2341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.708+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1117601866 shutdown_connections 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.708+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1117601866 >> 172.21.15.134:0/1117601866 conn(0x7fc2340b7ea0 msgr2=0x7fc2340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.708+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1117601866 shutdown_connections 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.709+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1117601866 wait complete. 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.709+0000 7fc23a0d9640 10 monclient: build_initial_monmap 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.709+0000 7fc23a0d9640 10 monclient: monmap: 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:00.710 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.710+0000 7fc23a0d9640 1 librados: starting msgr at 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.710+0000 7fc23a0d9640 1 librados: starting objecter 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.710+0000 7fc23a0d9640 1 Processor -- start 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.710+0000 7fc23a0d9640 1 -- start start 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.710+0000 7fc23a0d9640 1 librados: setting wanted keys 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.710+0000 7fc23a0d9640 1 librados: calling monclient init 2024-03-25T19:30:00.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.710+0000 7fc23a0d9640 10 monclient: init 2024-03-25T19:30:00.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:00.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:00.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc2340a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:00.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc23413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:00.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc2341430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:00.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340abd30 con 0x7fc2341430c0 2024-03-25T19:30:00.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2341d51f0 con 0x7fc23413f710 2024-03-25T19:30:00.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2341d77a0 con 0x7fc2340a98f0 2024-03-25T19:30:00.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:00.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.711+0000 7fc23a0d9640 10 monclient(hunting): authenticate will time out at 6032.869817s 2024-03-25T19:30:00.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.714+0000 7fc2337fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc2340a98f0 0x7fc234138740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:39168/0 (socket says 172.21.15.134:39168) 2024-03-25T19:30:00.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.714+0000 7fc2337fe640 1 -- 172.21.15.134:0/1622344653 learned_addr learned my addr 172.21.15.134:0/1622344653 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:00.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.715+0000 7fc21bfff640 10 client.?.objecter ms_handle_connect 0x7fc2340a98f0 2024-03-25T19:30:00.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.715+0000 7fc21bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.754+0000 7fc21bfff640 10 client.?.objecter ms_handle_connect 0x7fc23413f710 2024-03-25T19:30:00.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.754+0000 7fc21bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.761+0000 7fc21bfff640 10 client.?.objecter ms_handle_connect 0x7fc2341430c0 2024-03-25T19:30:00.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.761+0000 7fc21bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.786+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197302178 0 0) 0x7fc2340abd30 con 0x7fc2341430c0 2024-03-25T19:30:00.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.786+0000 7fc21bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.786+0000 7fc21bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.786+0000 7fc21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.786+0000 7fc21bfff640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:30:00.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.786+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc220004eb0 con 0x7fc2341430c0 2024-03-25T19:30:00.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.791+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483174216 0 0) 0x7fc2341d51f0 con 0x7fc23413f710 2024-03-25T19:30:00.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.791+0000 7fc21bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.791+0000 7fc21bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.791+0000 7fc21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.791+0000 7fc21bfff640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:30:00.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.791+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2340abd30 con 0x7fc23413f710 2024-03-25T19:30:00.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.792+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147939022 0 0) 0x7fc2341d77a0 con 0x7fc2340a98f0 2024-03-25T19:30:00.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.792+0000 7fc21bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.792+0000 7fc21bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.792+0000 7fc21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.792+0000 7fc21bfff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:30:00.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.792+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2341d51f0 con 0x7fc2340a98f0 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.797+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3037145297 0 0) 0x7fc220004eb0 con 0x7fc2341430c0 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.797+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2341d77a0 con 0x7fc2341430c0 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.797+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc22c001b30 con 0x7fc2341430c0 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.797+0000 7fc21bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.797+0000 7fc21bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.797+0000 7fc21bfff640 10 monclient(hunting): dump: 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.797 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.798 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:00.798 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:00.798 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:00.798 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:00.798 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:00.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.797+0000 7fc21bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.818+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1070939066 0 0) 0x7fc2341d77a0 con 0x7fc2341430c0 2024-03-25T19:30:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6790/0 conn(0x7fc2340a98f0 legacy=0x7fc234138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.159:6789/0 conn(0x7fc23413f710 legacy=0x7fc2341d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 1 monclient: found mon.a 2024-03-25T19:30:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234138e50 con 0x7fc2341430c0 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 10 monclient: _finish_auth 0 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc21bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 5 monclient: authenticate success, global_id 4311 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 10 monclient: _renew_subs 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2341d83d0 con 0x7fc2341430c0 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 10 monclient: _renew_subs 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2341381b0 con 0x7fc2341430c0 2024-03-25T19:30:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.819+0000 7fc23a0d9640 1 librados: init done 2024-03-25T19:30:00.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.820+0000 7fc23a0d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.820+0000 7fc23a0d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.820+0000 7fc23a0d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:00.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.820+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2341d7600 con 0x7fc2341430c0 2024-03-25T19:30:00.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc233fff640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6789/0 conn(0x7fc2341430c0 legacy=0x7fc2341e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:30:00.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc233fff640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6789/0 conn(0x7fc2341430c0 legacy=0x7fc2341e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:30:00.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc233fff640 1 --1- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6789/0 conn(0x7fc2341430c0 0x7fc2341e1e30 :-1 s=OPENED pgs=7559 cs=1 l=1).handle_message read tag failed 2024-03-25T19:30:00.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc233fff640 1 --1- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6789/0 conn(0x7fc2341430c0 0x7fc2341e1e30 :-1 s=OPENED pgs=7559 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:30:00.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 10 monclient: ms_handle_reset current mon v1:172.21.15.134:6789/0 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6789/0 conn(0x7fc2341430c0 legacy=0x7fc2341e1e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 10 monclient(hunting): picked mon.a con 0x7fc2200063d0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 10 monclient(hunting): picked mon.b con 0x7fc23413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 10 monclient(hunting): picked mon.c con 0x7fc2340a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc2341d77a0 con 0x7fc2200063d0 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc220004eb0 con 0x7fc23413f710 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc2200139d0 con 0x7fc2340a98f0 2024-03-25T19:30:00.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.868+0000 7fc21bfff640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:00.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.873+0000 7fc21bfff640 10 client.4311.objecter ms_handle_connect 0x7fc23413f710 2024-03-25T19:30:00.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.873+0000 7fc21bfff640 10 client.4311.objecter resend_mon_ops 2024-03-25T19:30:00.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.893+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743195736 0 0) 0x7fc220004eb0 con 0x7fc23413f710 2024-03-25T19:30:00.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.893+0000 7fc21bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.893+0000 7fc21bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.893+0000 7fc21bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:30:00.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.893+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.159:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc220012900 con 0x7fc23413f710 2024-03-25T19:30:00.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.903+0000 7fc21bfff640 10 client.4311.objecter ms_handle_connect 0x7fc2200063d0 2024-03-25T19:30:00.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.903+0000 7fc21bfff640 10 client.4311.objecter resend_mon_ops 2024-03-25T19:30:00.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 391571607 0 0) 0x7fc220012900 con 0x7fc23413f710 2024-03-25T19:30:00.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6790/0 conn(0x7fc2340a98f0 legacy=0x7fc22000fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.134:6789/0 conn(0x7fc2200063d0 legacy=0x7fc2200067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 1 monclient: found mon.b 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fc220014db0 con 0x7fc23413f710 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2200139d0 con 0x7fc23413f710 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: _finish_auth 0 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc228001580 con 0x7fc23413f710 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:00.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.906+0000 7fc21bfff640 10 monclient: dump: 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:00.908 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:00.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.915+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc228001880 con 0x7fc23413f710 2024-03-25T19:30:00.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.915+0000 7fc21bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:00.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:00.915+0000 7fc231ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:01.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.173+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fc228001ab0 con 0x7fc23413f710 2024-03-25T19:30:01.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.174+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fc228083100 con 0x7fc23413f710 2024-03-25T19:30:01.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.174+0000 7fc21bfff640 10 client.4311.objecter ms_dispatch 0x7fc234000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:01.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.174+0000 7fc21bfff640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:01.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.174+0000 7fc21bfff640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:01.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.174+0000 7fc2337fe640 10 monclient: get_auth_request con 0x7fc2341430c0 auth_method 0 2024-03-25T19:30:01.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.174+0000 7fc21bfff640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-25T19:30:01.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.174+0000 7fc21bfff640 10 client.4311.objecter ms_handle_connect 0x7fc2341430c0 2024-03-25T19:30:01.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.176+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc22804e9e0 con 0x7fc23413f710 2024-03-25T19:30:01.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.176+0000 7fc21bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:01.176 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.176+0000 7fc21bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:01.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.429+0000 7fc23a0d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:01.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.430+0000 7fc23a0d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:01.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.430+0000 7fc23a0d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:01.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.430+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2340627b0 con 0x7fc23413f710 2024-03-25T19:30:01.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.462+0000 7fc21bfff640 1 -- 172.21.15.134:0/1622344653 <== mon.1 v1:172.21.15.159:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fc228050910 con 0x7fc23413f710 2024-03-25T19:30:01.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.462+0000 7fc21bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:01.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.462+0000 7fc21bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:01.462 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:01.462 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:01.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 10 librados: watch_flush enter 2024-03-25T19:30:01.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 10 librados: watch_flush exit 2024-03-25T19:30:01.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 10 client.4311.objecter successfully canceled tick 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.159:6800/1875874887 conn(0x7fc2341430c0 legacy=0x7fc22007bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 10 monclient: shutdown 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.464+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 >> v1:172.21.15.159:6789/0 conn(0x7fc23413f710 legacy=0x7fc22000c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.465+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 shutdown_connections 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.465+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 >> 172.21.15.134:0/1622344653 conn(0x7fc2340b7ea0 msgr2=0x7fc234074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.465+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 shutdown_connections 2024-03-25T19:30:01.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.465+0000 7fc23a0d9640 1 -- 172.21.15.134:0/1622344653 wait complete. 2024-03-25T19:30:01.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:01.465+0000 7fc23a0d9640 1 librados: shutdown 2024-03-25T19:30:01.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:08.286 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:08.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.392+0000 7fcc84274640 10 monclient: get_monmap_and_config 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.392+0000 7fcc84274640 10 monclient: build_initial_monmap 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.393+0000 7fcc84274640 10 monclient: monmap: 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:08.394052+0000 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:08.394052+0000 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:08.393 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:08.394 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.393+0000 7fcc84274640 1 Processor -- start 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.393+0000 7fcc84274640 1 -- start start 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.393+0000 7fcc84274640 10 monclient: init 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7c0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 10 monclient(hunting): picked mon.noname-c con 0x7fcc7c13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 10 monclient(hunting): picked mon.noname-b con 0x7fcc7c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c142520 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0b28a0 con 0x7fcc7c13f710 2024-03-25T19:30:08.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0b2a30 con 0x7fcc7c1430c0 2024-03-25T19:30:08.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:08.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.394+0000 7fcc84274640 10 monclient(hunting): authenticate will time out at 6040.553133s 2024-03-25T19:30:08.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.395+0000 7fcc82811640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fcc7c1430c0 0x7fcc7c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:43764/0 (socket says 172.21.15.134:43764) 2024-03-25T19:30:08.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.395+0000 7fcc82811640 1 -- 172.21.15.134:0/2884158130 learned_addr learned my addr 172.21.15.134:0/2884158130 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:08.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.471+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771638159 0 0) 0x7fcc7c142520 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.471+0000 7fcc8100e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:08.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.471+0000 7fcc8100e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:08.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.471+0000 7fcc8100e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:08.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.471+0000 7fcc8100e640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:30:08.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.471+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68003520 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991856677 0 0) 0x7fcc68003520 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 >> v1:172.21.15.134:6790/0 conn(0x7fcc7c1430c0 legacy=0x7fcc7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 >> v1:172.21.15.159:6789/0 conn(0x7fcc7c13f710 legacy=0x7fcc7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 10 monclient: _finish_hunting 0 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 1 monclient: found mon.noname-a 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0b44c0 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 10 monclient: _finish_auth 0 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 10 monclient: _check_auth_tickets 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:08.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fcc78001690 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc8100e640 10 monclient: dump: 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc84274640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:30:08.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.484+0000 7fcc84274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:08.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.493+0000 7fcc8100e640 1 -- 172.21.15.134:0/2884158130 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc780017f0 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.493+0000 7fcc8100e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:08.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.493+0000 7fcc84274640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:08.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.493+0000 7fcc84274640 4 set_mon_vals no callback set 2024-03-25T19:30:08.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.493+0000 7fcc84274640 10 monclient: shutdown 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.494+0000 7fcc84274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.494+0000 7fcc84274640 1 -- 172.21.15.134:0/2884158130 >> v1:172.21.15.134:6789/0 conn(0x7fcc7c0a98f0 legacy=0x7fcc7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.494+0000 7fcc84274640 1 -- 172.21.15.134:0/2884158130 shutdown_connections 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.494+0000 7fcc84274640 1 -- 172.21.15.134:0/2884158130 >> 172.21.15.134:0/2884158130 conn(0x7fcc7c0b7ea0 msgr2=0x7fcc7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.494+0000 7fcc84274640 1 -- 172.21.15.134:0/2884158130 shutdown_connections 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.494+0000 7fcc84274640 1 -- 172.21.15.134:0/2884158130 wait complete. 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 10 monclient: build_initial_monmap 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 10 monclient: monmap: 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:08.494 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:08.495 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:08.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 1 librados: starting msgr at 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 1 librados: starting objecter 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 1 Processor -- start 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 1 -- start start 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 1 librados: setting wanted keys 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 1 librados: calling monclient init 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.495+0000 7fcc84274640 10 monclient: init 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 10 monclient(hunting): picked mon.noname-b con 0x7fcc7c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 10 monclient(hunting): picked mon.noname-c con 0x7fcc7c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c139870 con 0x7fcc7c1430c0 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c1d76a0 con 0x7fcc7c0a98f0 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c1d82d0 con 0x7fcc7c13f710 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:08.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.496+0000 7fcc84274640 10 monclient(hunting): authenticate will time out at 6040.655436s 2024-03-25T19:30:08.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.504+0000 7fcc82811640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fcc7c1430c0 0x7fcc7c134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:38998/0 (socket says 172.21.15.134:38998) 2024-03-25T19:30:08.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.504+0000 7fcc82811640 1 -- 172.21.15.134:0/3797644420 learned_addr learned my addr 172.21.15.134:0/3797644420 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:08.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.509+0000 7fcc72ffd640 10 client.?.objecter ms_handle_connect 0x7fcc7c1430c0 2024-03-25T19:30:08.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.509+0000 7fcc72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:08.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.513+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338019915 0 0) 0x7fcc7c139870 con 0x7fcc7c1430c0 2024-03-25T19:30:08.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.513+0000 7fcc72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:08.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.513+0000 7fcc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:08.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.513+0000 7fcc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:08.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.513+0000 7fcc72ffd640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:30:08.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.513+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60004eb0 con 0x7fcc7c1430c0 2024-03-25T19:30:08.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.517+0000 7fcc72ffd640 10 client.?.objecter ms_handle_connect 0x7fcc7c13f710 2024-03-25T19:30:08.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.517+0000 7fcc72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:08.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.522+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 814430681 0 0) 0x7fcc60004eb0 con 0x7fcc7c1430c0 2024-03-25T19:30:08.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.522+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc7c139870 con 0x7fcc7c1430c0 2024-03-25T19:30:08.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.522+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fcc74001b10 con 0x7fcc7c1430c0 2024-03-25T19:30:08.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.522+0000 7fcc72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:08.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.522+0000 7fcc72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.522+0000 7fcc72ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:08.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.522+0000 7fcc72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:08.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.523+0000 7fcc72ffd640 10 client.?.objecter ms_handle_connect 0x7fcc7c0a98f0 2024-03-25T19:30:08.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.523+0000 7fcc72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:08.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1105794064 0 0) 0x7fcc7c139870 con 0x7fcc7c1430c0 2024-03-25T19:30:08.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 >> v1:172.21.15.134:6790/0 conn(0x7fcc7c13f710 legacy=0x7fcc7c138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 >> v1:172.21.15.159:6789/0 conn(0x7fcc7c0a98f0 legacy=0x7fcc7c1d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 1 monclient: found mon.a 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c138c10 con 0x7fcc7c1430c0 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 10 monclient: _renew_subs 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c137cb0 con 0x7fcc7c1430c0 2024-03-25T19:30:08.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:08.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 10 monclient: _renew_subs 2024-03-25T19:30:08.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:08.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c134970 con 0x7fcc7c1430c0 2024-03-25T19:30:08.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.529+0000 7fcc84274640 1 librados: init done 2024-03-25T19:30:08.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.530+0000 7fcc84274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:08.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.530+0000 7fcc84274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:08.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.530+0000 7fcc84274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:08.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.530+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c1d7500 con 0x7fcc7c1430c0 2024-03-25T19:30:08.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.846+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc74002080 con 0x7fcc7c1430c0 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.846+0000 7fcc72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.846+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fcc74004b60 con 0x7fcc7c1430c0 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.846+0000 7fcc72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.846+0000 7fcc72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.846+0000 7fcc72ffd640 10 monclient: dump: 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:08.847 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:08.848 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:08.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.846+0000 7fcc8080d640 4 set_mon_vals no callback set 2024-03-25T19:30:08.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.847+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fcc74004d80 con 0x7fcc7c1430c0 2024-03-25T19:30:08.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.848+0000 7fcc82010640 10 monclient: get_auth_request con 0x7fcc6006dc00 auth_method 0 2024-03-25T19:30:08.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.848+0000 7fcc72ffd640 10 client.4317.objecter ms_handle_connect 0x7fcc6006dc00 2024-03-25T19:30:08.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fcc74054970 con 0x7fcc7c1430c0 2024-03-25T19:30:08.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 10 client.4317.objecter ms_dispatch 0x7fcc7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:08.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:08.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:08.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T19:30:08.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc740502b0 con 0x7fcc7c1430c0 2024-03-25T19:30:08.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:08.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:08.854+0000 7fcc72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:09.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.103+0000 7fcc84274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:09.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.103+0000 7fcc84274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:09.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.103+0000 7fcc84274640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:09.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.103+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc7c13f180 con 0x7fcc7c1430c0 2024-03-25T19:30:09.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.131+0000 7fcc72ffd640 1 -- 172.21.15.134:0/3797644420 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fcc7c13f180 con 0x7fcc7c1430c0 2024-03-25T19:30:09.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.131+0000 7fcc72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:09.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.131+0000 7fcc72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:09.131 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:09.131 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:09.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.133+0000 7fcc84274640 10 librados: watch_flush enter 2024-03-25T19:30:09.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 10 librados: watch_flush exit 2024-03-25T19:30:09.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T19:30:09.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 10 client.4317.objecter successfully canceled tick 2024-03-25T19:30:09.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 >> v1:172.21.15.159:6800/1875874887 conn(0x7fcc6006dc00 legacy=0x7fcc600700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:09.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 10 monclient: shutdown 2024-03-25T19:30:09.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:09.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 >> v1:172.21.15.134:6789/0 conn(0x7fcc7c1430c0 legacy=0x7fcc7c134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:09.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 shutdown_connections 2024-03-25T19:30:09.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 >> 172.21.15.134:0/3797644420 conn(0x7fcc7c0b7ea0 msgr2=0x7fcc7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:09.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 shutdown_connections 2024-03-25T19:30:09.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 1 -- 172.21.15.134:0/3797644420 wait complete. 2024-03-25T19:30:09.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:09.134+0000 7fcc84274640 1 librados: shutdown 2024-03-25T19:30:09.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:15.953 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:16.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.075+0000 7f70bfeaa640 10 monclient: get_monmap_and_config 2024-03-25T19:30:16.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.075+0000 7f70bfeaa640 10 monclient: build_initial_monmap 2024-03-25T19:30:16.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.075+0000 7f70bfeaa640 10 monclient: monmap: 2024-03-25T19:30:16.075 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:16.076721+0000 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:16.076721+0000 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:16.076 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.076+0000 7f70bfeaa640 1 Processor -- start 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.076+0000 7f70bfeaa640 1 -- start start 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.076+0000 7f70bfeaa640 10 monclient: init 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 10 monclient(hunting): picked mon.noname-c con 0x7f70b80a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 10 monclient(hunting): picked mon.noname-a con 0x7f70b813f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 10 monclient(hunting): picked mon.noname-b con 0x7f70b81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b8142520 con 0x7f70b813f710 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b80b28a0 con 0x7f70b80a98f0 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b80b2a30 con 0x7f70b81430c0 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:16.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.077+0000 7f70bfeaa640 10 monclient(hunting): authenticate will time out at 6048.236503s 2024-03-25T19:30:16.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.087+0000 7f70bdc46640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f70b80a98f0 0x7f70b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:39512/0 (socket says 172.21.15.134:39512) 2024-03-25T19:30:16.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.087+0000 7f70bdc46640 1 -- 172.21.15.134:0/3727721683 learned_addr learned my addr 172.21.15.134:0/3727721683 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:16.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960992350 0 0) 0x7f70b80b28a0 con 0x7f70b80a98f0 2024-03-25T19:30:16.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): _negotiate 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a4003520 con 0x7f70b80a98f0 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806917779 0 0) 0x7f70b80b2a30 con 0x7f70b81430c0 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): _negotiate 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:30:16.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.098+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b80b28a0 con 0x7f70b81430c0 2024-03-25T19:30:16.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1161174613 0 0) 0x7f70a4003520 con 0x7f70b80a98f0 2024-03-25T19:30:16.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 >> v1:172.21.15.134:6790/0 conn(0x7f70b81430c0 legacy=0x7f70b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 >> v1:172.21.15.134:6789/0 conn(0x7f70b813f710 legacy=0x7f70b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 10 monclient: _finish_hunting 0 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 1 monclient: found mon.noname-c 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b80b44c0 con 0x7f70b80a98f0 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 10 monclient: _finish_auth 0 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 10 monclient: _check_auth_tickets 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f70b4001690 con 0x7f70b80a98f0 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bcc44640 10 monclient: dump: 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:16.106 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bfeaa640 5 monclient: authenticate success, global_id 4297 2024-03-25T19:30:16.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.105+0000 7f70bfeaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:16.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b40017f0 con 0x7f70b80a98f0 2024-03-25T19:30:16.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bcc44640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:16.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bcc44640 1 -- 172.21.15.134:0/3727721683 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f70b4001c00 con 0x7f70b80a98f0 2024-03-25T19:30:16.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bfeaa640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:16.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bfeaa640 4 set_mon_vals no callback set 2024-03-25T19:30:16.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bcc44640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bcc44640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.121+0000 7f70bcc44640 10 monclient: dump: 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:16.122 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.122+0000 7f70bfeaa640 10 monclient: shutdown 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.122+0000 7f70bfeaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.122+0000 7f70bfeaa640 1 -- 172.21.15.134:0/3727721683 >> v1:172.21.15.159:6789/0 conn(0x7f70b80a98f0 legacy=0x7f70b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.122+0000 7f70bfeaa640 1 -- 172.21.15.134:0/3727721683 shutdown_connections 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.122+0000 7f70bfeaa640 1 -- 172.21.15.134:0/3727721683 >> 172.21.15.134:0/3727721683 conn(0x7f70b80b7ea0 msgr2=0x7f70b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.122+0000 7f70bfeaa640 1 -- 172.21.15.134:0/3727721683 shutdown_connections 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.122+0000 7f70bfeaa640 1 -- 172.21.15.134:0/3727721683 wait complete. 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.123+0000 7f70bfeaa640 10 monclient: build_initial_monmap 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.123+0000 7f70bfeaa640 10 monclient: monmap: 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:16.124 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.123+0000 7f70bfeaa640 1 librados: starting msgr at 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.123+0000 7f70bfeaa640 1 librados: starting objecter 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.123+0000 7f70bfeaa640 1 Processor -- start 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.124+0000 7f70bfeaa640 1 -- start start 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.124+0000 7f70bfeaa640 1 librados: setting wanted keys 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.124+0000 7f70bfeaa640 1 librados: calling monclient init 2024-03-25T19:30:16.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.124+0000 7f70bfeaa640 10 monclient: init 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.124+0000 7f70bfeaa640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.124+0000 7f70bfeaa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.124+0000 7f70bfeaa640 10 monclient(hunting): picked mon.noname-c con 0x7f70b80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.125+0000 7f70bfeaa640 10 monclient(hunting): picked mon.noname-a con 0x7f70b813f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.125+0000 7f70bfeaa640 10 monclient(hunting): picked mon.noname-b con 0x7f70b81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.125+0000 7f70bfeaa640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b8138b40 con 0x7f70b813f710 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.125+0000 7f70bfeaa640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b8138e20 con 0x7f70b81430c0 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.125+0000 7f70bfeaa640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b8139100 con 0x7f70b80a98f0 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.125+0000 7f70bfeaa640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:16.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.125+0000 7f70bfeaa640 10 monclient(hunting): authenticate will time out at 6048.284121s 2024-03-25T19:30:16.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.141+0000 7f70be447640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f70b81430c0 0x7f70b81e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:39546/0 (socket says 172.21.15.134:39546) 2024-03-25T19:30:16.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.141+0000 7f70be447640 1 -- 172.21.15.134:0/2152952614 learned_addr learned my addr 172.21.15.134:0/2152952614 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:16.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.143+0000 7f70ae7fc640 10 client.?.objecter ms_handle_connect 0x7f70b81430c0 2024-03-25T19:30:16.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.143+0000 7f70ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:16.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.144+0000 7f70ae7fc640 10 client.?.objecter ms_handle_connect 0x7f70b80a98f0 2024-03-25T19:30:16.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.144+0000 7f70ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:16.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.148+0000 7f70ae7fc640 10 client.?.objecter ms_handle_connect 0x7f70b813f710 2024-03-25T19:30:16.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.148+0000 7f70ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:16.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.152+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878704988 0 0) 0x7f70b8138e20 con 0x7f70b81430c0 2024-03-25T19:30:16.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.152+0000 7f70ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:16.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.152+0000 7f70ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:16.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.152+0000 7f70ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:16.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.152+0000 7f70ae7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:30:16.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.152+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c004eb0 con 0x7f70b81430c0 2024-03-25T19:30:16.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.159+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867265048 0 0) 0x7f70b8139100 con 0x7f70b80a98f0 2024-03-25T19:30:16.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.159+0000 7f70ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:16.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.159+0000 7f70ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:16.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.159+0000 7f70ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:16.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.159+0000 7f70ae7fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:30:16.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.159+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b8138e20 con 0x7f70b80a98f0 2024-03-25T19:30:16.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.162+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 665718700 0 0) 0x7f709c004eb0 con 0x7f70b81430c0 2024-03-25T19:30:16.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.162+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b8139100 con 0x7f70b81430c0 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.162+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f70b0001bf0 con 0x7f70b81430c0 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.162+0000 7f70ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.162+0000 7f70ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.162+0000 7f70ae7fc640 10 monclient(hunting): dump: 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:16.163 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:16.164 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.162+0000 7f70ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:16.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.164+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137745829 0 0) 0x7f70b8138b40 con 0x7f70b813f710 2024-03-25T19:30:16.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.164+0000 7f70ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:16.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.164+0000 7f70ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:16.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.164+0000 7f70ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:16.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.164+0000 7f70ae7fc640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:30:16.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.164+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c004eb0 con 0x7f70b813f710 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.166+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116933852 0 0) 0x7f70b8138e20 con 0x7f70b80a98f0 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.166+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b8138b40 con 0x7f70b80a98f0 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.166+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f70b4001bc0 con 0x7f70b80a98f0 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.166+0000 7f70ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.166+0000 7f70ae7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.166+0000 7f70ae7fc640 10 monclient(hunting): dump: 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.166 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:16.167 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:16.167 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:16.167 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:16.167 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:16.167 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.178+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1056831265 0 0) 0x7f709c004eb0 con 0x7f70b813f710 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.178+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b8138e20 con 0x7f70b813f710 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.178+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f70a8001870 con 0x7f70b813f710 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.178+0000 7f70ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.178+0000 7f70ae7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.178+0000 7f70ae7fc640 10 monclient(hunting): dump: 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.178 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.179 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:16.179 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:16.179 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:16.179 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:16.179 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:16.179 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3685799020 0 0) 0x7f70b8138b40 con 0x7f70b80a98f0 2024-03-25T19:30:16.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 >> v1:172.21.15.159:6789/0 conn(0x7f70b81430c0 legacy=0x7f70b81e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 >> v1:172.21.15.134:6789/0 conn(0x7f70b813f710 legacy=0x7f70b8137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 1 monclient: found mon.c 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b81397a0 con 0x7f70b80a98f0 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 5 monclient: authenticate success, global_id 4301 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 10 monclient: _renew_subs 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b8136aa0 con 0x7f70b80a98f0 2024-03-25T19:30:16.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:16.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 10 monclient: _renew_subs 2024-03-25T19:30:16.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:30:16.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b813c320 con 0x7f70b80a98f0 2024-03-25T19:30:16.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.184+0000 7f70bfeaa640 1 librados: init done 2024-03-25T19:30:16.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.185+0000 7f70bfeaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:16.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.185+0000 7f70bfeaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:16.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.185+0000 7f70bfeaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:30:16.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.185+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b81395b0 con 0x7f70b80a98f0 2024-03-25T19:30:16.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.494+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b4004a90 con 0x7f70b80a98f0 2024-03-25T19:30:16.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.494+0000 7f70ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.494+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f70b4004da0 con 0x7f70b80a98f0 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.494+0000 7f70ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.494+0000 7f70ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.494+0000 7f70ae7fc640 10 monclient: dump: 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:16.495 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:16.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.494+0000 7f70affff640 4 set_mon_vals no callback set 2024-03-25T19:30:16.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.495+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f70b401ad70 con 0x7f70b80a98f0 2024-03-25T19:30:16.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.496+0000 7f70bd445640 10 monclient: get_auth_request con 0x7f709c06e0b0 auth_method 0 2024-03-25T19:30:16.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.496+0000 7f70ae7fc640 10 client.4301.objecter ms_handle_connect 0x7f709c06e0b0 2024-03-25T19:30:16.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f70b4054860 con 0x7f70b80a98f0 2024-03-25T19:30:16.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 10 client.4301.objecter ms_dispatch 0x7f70b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:16.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:16.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:16.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-25T19:30:16.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70b4050370 con 0x7f70b80a98f0 2024-03-25T19:30:16.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:16.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.501+0000 7f70ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:16.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.720+0000 7f70bfeaa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:16.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.720+0000 7f70bfeaa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:16.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.720+0000 7f70bfeaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:30:16.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.720+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70b8139930 con 0x7f70b80a98f0 2024-03-25T19:30:16.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.782+0000 7f70ae7fc640 1 -- 172.21.15.134:0/2152952614 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f70b40522a0 con 0x7f70b80a98f0 2024-03-25T19:30:16.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.782+0000 7f70ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:16.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.782+0000 7f70ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:16.783 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:16.783 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:16.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 10 librados: watch_flush enter 2024-03-25T19:30:16.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 10 librados: watch_flush exit 2024-03-25T19:30:16.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 10 client.4301.objecter successfully canceled tick 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 >> v1:172.21.15.159:6800/1875874887 conn(0x7f709c06e0b0 legacy=0x7f709c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 10 monclient: shutdown 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 >> v1:172.21.15.134:6790/0 conn(0x7f70b80a98f0 legacy=0x7f70b8137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 shutdown_connections 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 >> 172.21.15.134:0/2152952614 conn(0x7f70b80b7ea0 msgr2=0x7f70b814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 shutdown_connections 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 1 -- 172.21.15.134:0/2152952614 wait complete. 2024-03-25T19:30:16.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:16.785+0000 7f70bfeaa640 1 librados: shutdown 2024-03-25T19:30:16.797 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:19.799 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:24.471 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.813 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:24.485 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.851 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:24.524 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.852 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:24.524 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.863 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:24.536 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.871 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:24.518 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.877 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:24.524 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.893 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:24.566 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:19.916 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:24.563 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:23.604 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:23.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.721+0000 7fe72d13b640 10 monclient: get_monmap_and_config 2024-03-25T19:30:23.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.721+0000 7fe72d13b640 10 monclient: build_initial_monmap 2024-03-25T19:30:23.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.721+0000 7fe72d13b640 10 monclient: monmap: 2024-03-25T19:30:23.721 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:23.722431+0000 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:23.722431+0000 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:23.722 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:23.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.722+0000 7fe72d13b640 1 Processor -- start 2024-03-25T19:30:23.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.722+0000 7fe72d13b640 1 -- start start 2024-03-25T19:30:23.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.722+0000 7fe72d13b640 10 monclient: init 2024-03-25T19:30:23.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:23.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:23.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 10 monclient(hunting): picked mon.noname-a con 0x7fe7280a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:23.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 10 monclient(hunting): picked mon.noname-c con 0x7fe72813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:23.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 10 monclient(hunting): picked mon.noname-b con 0x7fe7281430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:23.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe728142520 con 0x7fe7280a98f0 2024-03-25T19:30:23.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7280b28a0 con 0x7fe72813f710 2024-03-25T19:30:23.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7280b2a30 con 0x7fe7281430c0 2024-03-25T19:30:23.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:23.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.723+0000 7fe72d13b640 10 monclient(hunting): authenticate will time out at 6055.882204s 2024-03-25T19:30:23.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.726+0000 7fe71ffff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe72813f710 0x7fe7281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:48846/0 (socket says 172.21.15.134:48846) 2024-03-25T19:30:23.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.726+0000 7fe71ffff640 1 -- 172.21.15.134:0/2817934169 learned_addr learned my addr 172.21.15.134:0/2817934169 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:23.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.742+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655836112 0 0) 0x7fe7280b28a0 con 0x7fe72813f710 2024-03-25T19:30:23.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.742+0000 7fe72659c640 10 monclient(hunting): _negotiate 2024-03-25T19:30:23.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.742+0000 7fe72659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:23.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.742+0000 7fe72659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:23.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.742+0000 7fe72659c640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:30:23.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.742+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe714003520 con 0x7fe72813f710 2024-03-25T19:30:23.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.743+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663553117 0 0) 0x7fe7280b2a30 con 0x7fe7281430c0 2024-03-25T19:30:23.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.743+0000 7fe72659c640 10 monclient(hunting): _negotiate 2024-03-25T19:30:23.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.743+0000 7fe72659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:23.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.743+0000 7fe72659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:23.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.743+0000 7fe72659c640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:30:23.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.743+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7280b28a0 con 0x7fe7281430c0 2024-03-25T19:30:23.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831452686 0 0) 0x7fe714003520 con 0x7fe72813f710 2024-03-25T19:30:23.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 >> v1:172.21.15.134:6790/0 conn(0x7fe7281430c0 legacy=0x7fe72814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:23.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 >> v1:172.21.15.134:6789/0 conn(0x7fe7280a98f0 legacy=0x7fe7280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:23.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 10 monclient: _finish_hunting 0 2024-03-25T19:30:23.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 1 monclient: found mon.noname-c 2024-03-25T19:30:23.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:23.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7280b44c0 con 0x7fe72813f710 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 10 monclient: _finish_auth 0 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 10 monclient: _check_auth_tickets 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe710001690 con 0x7fe72813f710 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72659c640 10 monclient: dump: 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:23.752 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:23.753 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:23.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72d13b640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:30:23.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.751+0000 7fe72d13b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:23.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7100017f0 con 0x7fe72813f710 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72659c640 1 -- 172.21.15.134:0/2817934169 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe710001c40 con 0x7fe72813f710 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72659c640 10 monclient: dump: 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:23.757 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:23.758 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:23.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72d13b640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:23.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.757+0000 7fe72d13b640 4 set_mon_vals no callback set 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.758+0000 7fe72d13b640 10 monclient: shutdown 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.758+0000 7fe72d13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.758+0000 7fe72d13b640 1 -- 172.21.15.134:0/2817934169 >> v1:172.21.15.159:6789/0 conn(0x7fe72813f710 legacy=0x7fe7281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.758+0000 7fe72d13b640 1 -- 172.21.15.134:0/2817934169 shutdown_connections 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.758+0000 7fe72d13b640 1 -- 172.21.15.134:0/2817934169 >> 172.21.15.134:0/2817934169 conn(0x7fe7280b7ea0 msgr2=0x7fe7280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.758+0000 7fe72d13b640 1 -- 172.21.15.134:0/2817934169 shutdown_connections 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.758+0000 7fe72d13b640 1 -- 172.21.15.134:0/2817934169 wait complete. 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.759+0000 7fe72d13b640 10 monclient: build_initial_monmap 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.759+0000 7fe72d13b640 10 monclient: monmap: 2024-03-25T19:30:23.759 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:23.760 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.759+0000 7fe72d13b640 1 librados: starting msgr at 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.759+0000 7fe72d13b640 1 librados: starting objecter 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.760+0000 7fe72d13b640 1 Processor -- start 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.760+0000 7fe72d13b640 1 -- start start 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.760+0000 7fe72d13b640 1 librados: setting wanted keys 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.760+0000 7fe72d13b640 1 librados: calling monclient init 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.760+0000 7fe72d13b640 10 monclient: init 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.760+0000 7fe72d13b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.760+0000 7fe72d13b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 10 monclient(hunting): picked mon.noname-a con 0x7fe7280a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 10 monclient(hunting): picked mon.noname-b con 0x7fe72813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 10 monclient(hunting): picked mon.noname-c con 0x7fe7281430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:23.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7281339d0 con 0x7fe7280a98f0 2024-03-25T19:30:23.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7281385a0 con 0x7fe72813f710 2024-03-25T19:30:23.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe728139340 con 0x7fe7281430c0 2024-03-25T19:30:23.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:23.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.761+0000 7fe72d13b640 10 monclient(hunting): authenticate will time out at 6055.920033s 2024-03-25T19:30:23.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.769+0000 7fe726d9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe7280a98f0 0x7fe7281dab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:48250/0 (socket says 172.21.15.134:48250) 2024-03-25T19:30:23.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.769+0000 7fe726d9d640 1 -- 172.21.15.134:0/816000134 learned_addr learned my addr 172.21.15.134:0/816000134 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:23.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.770+0000 7fe71f7fe640 10 client.?.objecter ms_handle_connect 0x7fe7280a98f0 2024-03-25T19:30:23.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.770+0000 7fe71f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:23.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.776+0000 7fe71f7fe640 10 client.?.objecter ms_handle_connect 0x7fe72813f710 2024-03-25T19:30:23.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.776+0000 7fe71f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:23.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.782+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846392473 0 0) 0x7fe7281339d0 con 0x7fe7280a98f0 2024-03-25T19:30:23.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.782+0000 7fe71f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:23.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.782+0000 7fe71f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:23.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.782+0000 7fe71f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:23.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.782+0000 7fe71f7fe640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:30:23.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.782+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe708004eb0 con 0x7fe7280a98f0 2024-03-25T19:30:23.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.791+0000 7fe71f7fe640 10 client.?.objecter ms_handle_connect 0x7fe7281430c0 2024-03-25T19:30:23.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.791+0000 7fe71f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.792+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277019907 0 0) 0x7fe708004eb0 con 0x7fe7280a98f0 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.792+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7281339d0 con 0x7fe7280a98f0 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.792+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe718001a30 con 0x7fe7280a98f0 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.792+0000 7fe71f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.792+0000 7fe71f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.792+0000 7fe71f7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:23.792 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:23.793 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:23.793 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:23.793 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:23.793 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:23.793 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:23.793 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:23.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.792+0000 7fe71f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:23.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.796+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461151640 0 0) 0x7fe7281385a0 con 0x7fe72813f710 2024-03-25T19:30:23.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.796+0000 7fe71f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:23.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.796+0000 7fe71f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:23.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.796+0000 7fe71f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:23.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.796+0000 7fe71f7fe640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:30:23.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.796+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe708004eb0 con 0x7fe72813f710 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1331046349 0 0) 0x7fe7281339d0 con 0x7fe7280a98f0 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 >> v1:172.21.15.134:6790/0 conn(0x7fe7281430c0 legacy=0x7fe7281332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 >> v1:172.21.15.159:6789/0 conn(0x7fe72813f710 legacy=0x7fe728132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 1 monclient: found mon.a 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728138080 con 0x7fe7280a98f0 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.803+0000 7fe71f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:23.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 5 monclient: authenticate success, global_id 4329 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 10 monclient: _renew_subs 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe728137610 con 0x7fe7280a98f0 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 10 monclient: _renew_subs 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe728137cb0 con 0x7fe7280a98f0 2024-03-25T19:30:23.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.804+0000 7fe72d13b640 1 librados: init done 2024-03-25T19:30:23.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.805+0000 7fe72d13b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:23.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.805+0000 7fe72d13b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:23.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.805+0000 7fe72d13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:23.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:23.805+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7280627b0 con 0x7fe7280a98f0 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.119+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe718001cb0 con 0x7fe7280a98f0 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.119+0000 7fe71f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.119+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe718003a60 con 0x7fe7280a98f0 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.119+0000 7fe71f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.119+0000 7fe71f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.119+0000 7fe71f7fe640 10 monclient: dump: 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:24.119 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:24.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.119+0000 7fe725d9b640 4 set_mon_vals no callback set 2024-03-25T19:30:24.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.120+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fe718003c40 con 0x7fe7280a98f0 2024-03-25T19:30:24.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.121+0000 7fe71ffff640 10 monclient: get_auth_request con 0x7fe70806deb0 auth_method 0 2024-03-25T19:30:24.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.121+0000 7fe71f7fe640 10 client.4329.objecter ms_handle_connect 0x7fe70806deb0 2024-03-25T19:30:24.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fe718054820 con 0x7fe7280a98f0 2024-03-25T19:30:24.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 10 client.4329.objecter ms_dispatch 0x7fe728000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:24.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:24.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:24.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-25T19:30:24.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe718050330 con 0x7fe7280a98f0 2024-03-25T19:30:24.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:24.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.125+0000 7fe71f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:24.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.355+0000 7fe72d13b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:24.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.355+0000 7fe72d13b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:24.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.355+0000 7fe72d13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:24.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.355+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe728138210 con 0x7fe7280a98f0 2024-03-25T19:30:24.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.382+0000 7fe71f7fe640 1 -- 172.21.15.134:0/816000134 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fe718052260 con 0x7fe7280a98f0 2024-03-25T19:30:24.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.382+0000 7fe71f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:24.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.382+0000 7fe71f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:24.382 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:24.383 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:24.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 10 librados: watch_flush enter 2024-03-25T19:30:24.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 10 librados: watch_flush exit 2024-03-25T19:30:24.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 10 client.4329.objecter successfully canceled tick 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 >> v1:172.21.15.159:6800/1875874887 conn(0x7fe70806deb0 legacy=0x7fe708070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 10 monclient: shutdown 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 >> v1:172.21.15.134:6789/0 conn(0x7fe7280a98f0 legacy=0x7fe7281dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 shutdown_connections 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 >> 172.21.15.134:0/816000134 conn(0x7fe7280b7ea0 msgr2=0x7fe72813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 shutdown_connections 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 1 -- 172.21.15.134:0/816000134 wait complete. 2024-03-25T19:30:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:24.384+0000 7fe72d13b640 1 librados: shutdown 2024-03-25T19:30:24.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:31.200 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:31.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.316+0000 7fe977b96640 10 monclient: get_monmap_and_config 2024-03-25T19:30:31.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.316+0000 7fe977b96640 10 monclient: build_initial_monmap 2024-03-25T19:30:31.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.316+0000 7fe977b96640 10 monclient: monmap: 2024-03-25T19:30:31.316 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:31.316 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:31.316 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:31.317537+0000 2024-03-25T19:30:31.317 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:31.317537+0000 2024-03-25T19:30:31.317 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:31.317 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:31.317 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:31.317 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:31.317 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:31.317 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:31.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.317+0000 7fe977b96640 1 Processor -- start 2024-03-25T19:30:31.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.317+0000 7fe977b96640 1 -- start start 2024-03-25T19:30:31.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.317+0000 7fe977b96640 10 monclient: init 2024-03-25T19:30:31.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:31.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 10 monclient(hunting): picked mon.noname-b con 0x7fe9700a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 10 monclient(hunting): picked mon.noname-a con 0x7fe97013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 10 monclient(hunting): picked mon.noname-c con 0x7fe9701430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe970142520 con 0x7fe97013f710 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700b28a0 con 0x7fe9701430c0 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700b2a30 con 0x7fe9700a98f0 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:31.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.318+0000 7fe977b96640 10 monclient(hunting): authenticate will time out at 6063.477123s 2024-03-25T19:30:31.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.324+0000 7fe975932640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe9700a98f0 0x7fe9700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:45980/0 (socket says 172.21.15.134:45980) 2024-03-25T19:30:31.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.324+0000 7fe975932640 1 -- 172.21.15.134:0/718045419 learned_addr learned my addr 172.21.15.134:0/718045419 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:31.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.350+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398468677 0 0) 0x7fe970142520 con 0x7fe97013f710 2024-03-25T19:30:31.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.350+0000 7fe974930640 10 monclient(hunting): _negotiate 2024-03-25T19:30:31.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.350+0000 7fe974930640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:31.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.350+0000 7fe974930640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:31.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.350+0000 7fe974930640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:30:31.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.350+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe95c003520 con 0x7fe97013f710 2024-03-25T19:30:31.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2688915666 0 0) 0x7fe95c003520 con 0x7fe97013f710 2024-03-25T19:30:31.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 >> v1:172.21.15.134:6790/0 conn(0x7fe9700a98f0 legacy=0x7fe9700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:31.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 >> v1:172.21.15.159:6789/0 conn(0x7fe9701430c0 legacy=0x7fe97014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 10 monclient: _finish_hunting 0 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 1 monclient: found mon.noname-a 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9700b44c0 con 0x7fe97013f710 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 10 monclient: _finish_auth 0 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 10 monclient: _check_auth_tickets 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe960001690 con 0x7fe97013f710 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe974930640 10 monclient: dump: 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:31.381 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe977b96640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:30:31.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.380+0000 7fe977b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:31.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9600017f0 con 0x7fe97013f710 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe974930640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe974930640 1 -- 172.21.15.134:0/718045419 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe960001c40 con 0x7fe97013f710 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe977b96640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe977b96640 4 set_mon_vals no callback set 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe974930640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe974930640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe974930640 10 monclient: dump: 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:31.443 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe977b96640 10 monclient: shutdown 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe977b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe977b96640 1 -- 172.21.15.134:0/718045419 >> v1:172.21.15.134:6789/0 conn(0x7fe97013f710 legacy=0x7fe9701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe977b96640 1 -- 172.21.15.134:0/718045419 shutdown_connections 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.442+0000 7fe977b96640 1 -- 172.21.15.134:0/718045419 >> 172.21.15.134:0/718045419 conn(0x7fe9700b7ea0 msgr2=0x7fe9700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.443+0000 7fe977b96640 1 -- 172.21.15.134:0/718045419 shutdown_connections 2024-03-25T19:30:31.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.443+0000 7fe977b96640 1 -- 172.21.15.134:0/718045419 wait complete. 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.443+0000 7fe977b96640 10 monclient: build_initial_monmap 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.443+0000 7fe977b96640 10 monclient: monmap: 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:31.446 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.444+0000 7fe977b96640 1 librados: starting msgr at 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.444+0000 7fe977b96640 1 librados: starting objecter 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.444+0000 7fe977b96640 1 Processor -- start 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.444+0000 7fe977b96640 1 -- start start 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.444+0000 7fe977b96640 1 librados: setting wanted keys 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.444+0000 7fe977b96640 1 librados: calling monclient init 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.444+0000 7fe977b96640 10 monclient: init 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:31.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 10 monclient(hunting): picked mon.noname-b con 0x7fe9700a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:31.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 10 monclient(hunting): picked mon.noname-a con 0x7fe97013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:31.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 10 monclient(hunting): picked mon.noname-c con 0x7fe9701430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:31.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe970139a80 con 0x7fe97013f710 2024-03-25T19:30:31.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe970137450 con 0x7fe9700a98f0 2024-03-25T19:30:31.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9701d7ba0 con 0x7fe9701430c0 2024-03-25T19:30:31.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:31.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.445+0000 7fe977b96640 10 monclient(hunting): authenticate will time out at 6063.604132s 2024-03-25T19:30:31.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.454+0000 7fe975932640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe9700a98f0 0x7fe970139370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58152/0 (socket says 172.21.15.134:58152) 2024-03-25T19:30:31.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.454+0000 7fe975932640 1 -- 172.21.15.134:0/3046328636 learned_addr learned my addr 172.21.15.134:0/3046328636 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:31.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.455+0000 7fe9667fc640 10 client.?.objecter ms_handle_connect 0x7fe9700a98f0 2024-03-25T19:30:31.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.455+0000 7fe9667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:31.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.466+0000 7fe9667fc640 10 client.?.objecter ms_handle_connect 0x7fe97013f710 2024-03-25T19:30:31.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.466+0000 7fe9667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:31.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.468+0000 7fe9667fc640 10 client.?.objecter ms_handle_connect 0x7fe9701430c0 2024-03-25T19:30:31.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.468+0000 7fe9667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:31.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.473+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714697405 0 0) 0x7fe970137450 con 0x7fe9700a98f0 2024-03-25T19:30:31.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.473+0000 7fe9667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:31.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.473+0000 7fe9667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:31.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.473+0000 7fe9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:31.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.473+0000 7fe9667fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:30:31.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.474+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe954004eb0 con 0x7fe9700a98f0 2024-03-25T19:30:31.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.477+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186882 0 0) 0x7fe9701d7ba0 con 0x7fe9701430c0 2024-03-25T19:30:31.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.477+0000 7fe9667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:31.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.477+0000 7fe9667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:31.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.477+0000 7fe9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:31.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.477+0000 7fe9667fc640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:30:31.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.477+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe970137450 con 0x7fe9701430c0 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.487+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570543428 0 0) 0x7fe970137450 con 0x7fe9701430c0 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.487+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9701d7ba0 con 0x7fe9701430c0 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.487+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe968001b00 con 0x7fe9701430c0 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.487+0000 7fe9667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.487+0000 7fe9667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.487+0000 7fe9667fc640 10 monclient(hunting): dump: 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:31.487 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:31.488 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:31.488 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:31.488 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:31.488 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:31.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.487+0000 7fe9667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2237627317 0 0) 0x7fe954004eb0 con 0x7fe9700a98f0 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe970137450 con 0x7fe9700a98f0 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe96c0018b0 con 0x7fe9700a98f0 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 10 monclient(hunting): dump: 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.489 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120221994 0 0) 0x7fe9700b44c0 con 0x7fe97013f710 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:30:31.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.489+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe954004eb0 con 0x7fe97013f710 2024-03-25T19:30:31.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2769900413 0 0) 0x7fe970137450 con 0x7fe9700a98f0 2024-03-25T19:30:31.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 >> v1:172.21.15.134:6790/0 conn(0x7fe9701430c0 legacy=0x7fe970132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:31.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 >> v1:172.21.15.134:6789/0 conn(0x7fe97013f710 legacy=0x7fe9701d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:31.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:31.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 1 monclient: found mon.b 2024-03-25T19:30:31.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:31.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9701368f0 con 0x7fe9700a98f0 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe9667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 10 monclient: _renew_subs 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe970136b40 con 0x7fe9700a98f0 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 10 monclient: _renew_subs 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe970137c00 con 0x7fe9700a98f0 2024-03-25T19:30:31.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.499+0000 7fe977b96640 1 librados: init done 2024-03-25T19:30:31.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.500+0000 7fe977b96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:31.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.500+0000 7fe977b96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:31.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.500+0000 7fe977b96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:31.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.500+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9700627b0 con 0x7fe9700a98f0 2024-03-25T19:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.782+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe96c0047e0 con 0x7fe9700a98f0 2024-03-25T19:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.782+0000 7fe9667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.782+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe96c004a90 con 0x7fe9700a98f0 2024-03-25T19:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.782+0000 7fe9667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.782+0000 7fe9667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.782+0000 7fe9667fc640 10 monclient: dump: 2024-03-25T19:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:31.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.782+0000 7fe967fff640 4 set_mon_vals no callback set 2024-03-25T19:30:31.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.783+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fe96c004cb0 con 0x7fe9700a98f0 2024-03-25T19:30:31.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.784+0000 7fe975131640 10 monclient: get_auth_request con 0x7fe95406df30 auth_method 0 2024-03-25T19:30:31.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.784+0000 7fe9667fc640 10 client.4309.objecter ms_handle_connect 0x7fe95406df30 2024-03-25T19:30:31.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.788+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fe96c054670 con 0x7fe9700a98f0 2024-03-25T19:30:31.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.788+0000 7fe9667fc640 10 client.4309.objecter ms_dispatch 0x7fe970000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:31.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.788+0000 7fe9667fc640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:31.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.788+0000 7fe9667fc640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:31.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.788+0000 7fe9667fc640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T19:30:31.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.789+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe96c0b18b0 con 0x7fe9700a98f0 2024-03-25T19:30:31.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.789+0000 7fe9667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:31.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:31.789+0000 7fe9667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:32.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.044+0000 7fe977b96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:32.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.044+0000 7fe977b96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:32.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.044+0000 7fe977b96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:32.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.044+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe970074cc0 con 0x7fe9700a98f0 2024-03-25T19:30:32.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.082+0000 7fe9667fc640 1 -- 172.21.15.134:0/3046328636 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fe96c050180 con 0x7fe9700a98f0 2024-03-25T19:30:32.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.082+0000 7fe9667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:32.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.082+0000 7fe9667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:32.082 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:32.082 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:32.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.083+0000 7fe977b96640 10 librados: watch_flush enter 2024-03-25T19:30:32.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 10 librados: watch_flush exit 2024-03-25T19:30:32.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T19:30:32.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 10 client.4309.objecter successfully canceled tick 2024-03-25T19:30:32.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 >> v1:172.21.15.159:6800/1875874887 conn(0x7fe95406df30 legacy=0x7fe9540703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 10 monclient: shutdown 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 >> v1:172.21.15.159:6789/0 conn(0x7fe9700a98f0 legacy=0x7fe970139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 shutdown_connections 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 >> 172.21.15.134:0/3046328636 conn(0x7fe9700b7ea0 msgr2=0x7fe970074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 shutdown_connections 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 1 -- 172.21.15.134:0/3046328636 wait complete. 2024-03-25T19:30:32.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:32.084+0000 7fe977b96640 1 librados: shutdown 2024-03-25T19:30:32.096 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:38.920 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:39.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.035+0000 7f8108043640 10 monclient: get_monmap_and_config 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.035+0000 7f8108043640 10 monclient: build_initial_monmap 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.035+0000 7f8108043640 10 monclient: monmap: 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:39.036088+0000 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:39.036088+0000 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:39.035 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.035+0000 7f8108043640 1 Processor -- start 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.036+0000 7f8108043640 1 -- start start 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.036+0000 7f8108043640 10 monclient: init 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.036+0000 7f8108043640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.036+0000 7f8108043640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.036+0000 7f8108043640 10 monclient(hunting): picked mon.noname-a con 0x7f81000a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.036+0000 7f8108043640 10 monclient(hunting): picked mon.noname-b con 0x7f810013f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.036+0000 7f8108043640 10 monclient(hunting): picked mon.noname-c con 0x7f81001430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.037+0000 7f8108043640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8100142520 con 0x7f81000a98f0 2024-03-25T19:30:39.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.037+0000 7f8108043640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000b28a0 con 0x7f81001430c0 2024-03-25T19:30:39.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.037+0000 7f8108043640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000b2a30 con 0x7f810013f710 2024-03-25T19:30:39.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.037+0000 7f8108043640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:39.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.037+0000 7f8108043640 10 monclient(hunting): authenticate will time out at 6071.196026s 2024-03-25T19:30:39.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.047+0000 7f81065e0640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 0x7f810014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34580/0 (socket says 172.21.15.134:34580) 2024-03-25T19:30:39.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.047+0000 7f81065e0640 1 -- 172.21.15.134:0/981060477 learned_addr learned my addr 172.21.15.134:0/981060477 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:39.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.065+0000 7f81065e0640 1 -- 172.21.15.134:0/981060477 >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 legacy=0x7f810014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:30:39.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.065+0000 7f81065e0640 1 -- 172.21.15.134:0/981060477 >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 legacy=0x7f810014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:30:39.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.065+0000 7f81065e0640 1 --1- 172.21.15.134:0/981060477 >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 0x7f810014dc30 :-1 s=OPENED pgs=12668 cs=1 l=1).handle_message read tag failed 2024-03-25T19:30:39.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.065+0000 7f81065e0640 1 --1- 172.21.15.134:0/981060477 >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 0x7f810014dc30 :-1 s=OPENED pgs=12668 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:30:39.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.065+0000 7f8104ddd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.159:6789/0 2024-03-25T19:30:39.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.066+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556364560 0 0) 0x7f8100142520 con 0x7f81000a98f0 2024-03-25T19:30:39.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.066+0000 7f8104ddd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:39.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.066+0000 7f8104ddd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:39.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.066+0000 7f8104ddd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:39.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.066+0000 7f8104ddd640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:30:39.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.066+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80ec003520 con 0x7f81000a98f0 2024-03-25T19:30:39.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.079+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250352201 0 0) 0x7f81000b2a30 con 0x7f810013f710 2024-03-25T19:30:39.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.079+0000 7f8104ddd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:39.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.079+0000 7f8104ddd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:39.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.079+0000 7f8104ddd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:39.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.079+0000 7f8104ddd640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:30:39.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.079+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8100142520 con 0x7f810013f710 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673457102 0 0) 0x7f80ec003520 con 0x7f81000a98f0 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 >> v1:172.21.15.134:6790/0 conn(0x7f810013f710 legacy=0x7f81001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 legacy=0x7f810014dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 1 monclient: found mon.noname-a 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:30:39.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000b44c0 con 0x7f81000a98f0 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 10 monclient: _finish_auth 0 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80fc001690 con 0x7f81000a98f0 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8104ddd640 10 monclient: dump: 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:39.088 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:39.089 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:39.089 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:39.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8108043640 5 monclient: authenticate success, global_id 4338 2024-03-25T19:30:39.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.087+0000 7f8108043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:39.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80fc0017f0 con 0x7f81000a98f0 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8104ddd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8104ddd640 1 -- 172.21.15.134:0/981060477 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80fc001c00 con 0x7f81000a98f0 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8108043640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8108043640 4 set_mon_vals no callback set 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8104ddd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8104ddd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.110+0000 7f8104ddd640 10 monclient: dump: 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:39.110 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:39.111 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:39.111 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:39.111 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:39.111 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:39.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.111+0000 7f8108043640 10 monclient: shutdown 2024-03-25T19:30:39.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.111+0000 7f8108043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:39.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.111+0000 7f8108043640 1 -- 172.21.15.134:0/981060477 >> v1:172.21.15.134:6789/0 conn(0x7f81000a98f0 legacy=0x7f81000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:39.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.111+0000 7f8108043640 1 -- 172.21.15.134:0/981060477 shutdown_connections 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.111+0000 7f8108043640 1 -- 172.21.15.134:0/981060477 >> 172.21.15.134:0/981060477 conn(0x7f81000b7ea0 msgr2=0x7f81000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.111+0000 7f8108043640 1 -- 172.21.15.134:0/981060477 shutdown_connections 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.111+0000 7f8108043640 1 -- 172.21.15.134:0/981060477 wait complete. 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 10 monclient: build_initial_monmap 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 10 monclient: monmap: 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:39.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 1 librados: starting msgr at 2024-03-25T19:30:39.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 1 librados: starting objecter 2024-03-25T19:30:39.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 1 Processor -- start 2024-03-25T19:30:39.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 1 -- start start 2024-03-25T19:30:39.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 1 librados: setting wanted keys 2024-03-25T19:30:39.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 1 librados: calling monclient init 2024-03-25T19:30:39.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.112+0000 7f8108043640 10 monclient: init 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 10 monclient(hunting): picked mon.noname-c con 0x7f81000a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 10 monclient(hunting): picked mon.noname-a con 0x7f810013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 10 monclient(hunting): picked mon.noname-b con 0x7f81001430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000b1710 con 0x7f810013f710 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000ad1f0 con 0x7f81001430c0 2024-03-25T19:30:39.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81001d7b70 con 0x7f81000a98f0 2024-03-25T19:30:39.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:39.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.113+0000 7f8108043640 10 monclient(hunting): authenticate will time out at 6071.272337s 2024-03-25T19:30:39.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.121+0000 7f81065e0640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 0x7f81001d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34648/0 (socket says 172.21.15.134:34648) 2024-03-25T19:30:39.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.121+0000 7f81065e0640 1 -- 172.21.15.134:0/2728699772 learned_addr learned my addr 172.21.15.134:0/2728699772 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:39.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.122+0000 7f80f67fc640 10 client.?.objecter ms_handle_connect 0x7f81000a98f0 2024-03-25T19:30:39.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.122+0000 7f80f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:39.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.125+0000 7f80f67fc640 10 client.?.objecter ms_handle_connect 0x7f81001430c0 2024-03-25T19:30:39.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.125+0000 7f80f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:39.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.127+0000 7f80f67fc640 10 client.?.objecter ms_handle_connect 0x7f810013f710 2024-03-25T19:30:39.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.127+0000 7f80f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:39.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.128+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603334880 0 0) 0x7f81001d7b70 con 0x7f81000a98f0 2024-03-25T19:30:39.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.128+0000 7f80f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:39.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.128+0000 7f80f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:39.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.128+0000 7f80f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:39.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.128+0000 7f80f67fc640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:30:39.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.128+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e4004eb0 con 0x7f81000a98f0 2024-03-25T19:30:39.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.135+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209871425 0 0) 0x7f81000b1710 con 0x7f810013f710 2024-03-25T19:30:39.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.135+0000 7f80f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:39.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.135+0000 7f80f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:39.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.135+0000 7f80f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:39.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.135+0000 7f80f67fc640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:30:39.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.135+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81001d7b70 con 0x7f810013f710 2024-03-25T19:30:39.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.137+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380654119 0 0) 0x7f81000ad1f0 con 0x7f81001430c0 2024-03-25T19:30:39.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.137+0000 7f80f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:39.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.137+0000 7f80f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:39.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.137+0000 7f80f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:39.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.137+0000 7f80f67fc640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:30:39.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.137+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81000b1710 con 0x7f81001430c0 2024-03-25T19:30:39.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.149+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1187748849 0 0) 0x7f81001d7b70 con 0x7f810013f710 2024-03-25T19:30:39.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.149+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81000ad1f0 con 0x7f810013f710 2024-03-25T19:30:39.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.149+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80f00018b0 con 0x7f810013f710 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.149+0000 7f80f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.149+0000 7f80f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.149+0000 7f80f67fc640 10 monclient(hunting): dump: 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:39.150 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:39.151 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:39.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.149+0000 7f80f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.158+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 116962734 0 0) 0x7f81000b1710 con 0x7f81001430c0 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.158+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81001d7b70 con 0x7f81001430c0 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.158+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80f8001b90 con 0x7f81001430c0 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.158+0000 7f80f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.158+0000 7f80f67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.158+0000 7f80f67fc640 10 monclient(hunting): dump: 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:39.158 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:39.159 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:39.159 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:39.159 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3759171616 0 0) 0x7f81000ad1f0 con 0x7f810013f710 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 >> v1:172.21.15.134:6790/0 conn(0x7f81000a98f0 legacy=0x7f81000b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 >> v1:172.21.15.159:6789/0 conn(0x7f81001430c0 legacy=0x7f81001d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 1 monclient: found mon.a 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:39.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81001d5000 con 0x7f810013f710 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f80f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 5 monclient: authenticate success, global_id 4341 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 10 monclient: _renew_subs 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81001d51f0 con 0x7f810013f710 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 10 monclient: _renew_subs 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81001d7820 con 0x7f810013f710 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.159+0000 7f8108043640 1 librados: init done 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.160+0000 7f8108043640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.160+0000 7f8108043640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.160+0000 7f8108043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:39.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.160+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81000627b0 con 0x7f810013f710 2024-03-25T19:30:39.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.497+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f00035f0 con 0x7f810013f710 2024-03-25T19:30:39.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.497+0000 7f80f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:39.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.497+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f80f0003900 con 0x7f810013f710 2024-03-25T19:30:39.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.497+0000 7f80f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:39.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.497+0000 7f80f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:39.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.497+0000 7f80f67fc640 10 monclient: dump: 2024-03-25T19:30:39.497 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:39.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.497+0000 7f80f7fff640 4 set_mon_vals no callback set 2024-03-25T19:30:39.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.498+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f80f00198d0 con 0x7f810013f710 2024-03-25T19:30:39.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.498+0000 7f8105ddf640 10 monclient: get_auth_request con 0x7f80e406df30 auth_method 0 2024-03-25T19:30:39.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.498+0000 7f80f67fc640 10 client.4341.objecter ms_handle_connect 0x7f80e406df30 2024-03-25T19:30:39.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f80f00535e0 con 0x7f810013f710 2024-03-25T19:30:39.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 10 client.4341.objecter ms_dispatch 0x7f8100000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:39.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:39.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:39.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-25T19:30:39.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80f004ef20 con 0x7f810013f710 2024-03-25T19:30:39.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:39.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.503+0000 7f80f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:39.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.758+0000 7f8108043640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:39.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.758+0000 7f8108043640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:39.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.758+0000 7f8108043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:39.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.758+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81001d7d00 con 0x7f810013f710 2024-03-25T19:30:39.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.804+0000 7f80f67fc640 1 -- 172.21.15.134:0/2728699772 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f80f0051020 con 0x7f810013f710 2024-03-25T19:30:39.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.804+0000 7f80f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:39.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.804+0000 7f80f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:39.805 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:39.805 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:39.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.806+0000 7f8108043640 10 librados: watch_flush enter 2024-03-25T19:30:39.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.806+0000 7f8108043640 10 librados: watch_flush exit 2024-03-25T19:30:39.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.806+0000 7f8108043640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-25T19:30:39.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.806+0000 7f8108043640 10 client.4341.objecter successfully canceled tick 2024-03-25T19:30:39.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.806+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 >> v1:172.21.15.159:6800/1875874887 conn(0x7f80e406df30 legacy=0x7f80e40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:39.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 10 monclient: shutdown 2024-03-25T19:30:39.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:39.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 >> v1:172.21.15.134:6789/0 conn(0x7f810013f710 legacy=0x7f81001d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:39.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 shutdown_connections 2024-03-25T19:30:39.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 >> 172.21.15.134:0/2728699772 conn(0x7f81000b7ea0 msgr2=0x7f810013ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:39.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 shutdown_connections 2024-03-25T19:30:39.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 1 -- 172.21.15.134:0/2728699772 wait complete. 2024-03-25T19:30:39.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:39.807+0000 7f8108043640 1 librados: shutdown 2024-03-25T19:30:39.816 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:39.850 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:44.522 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.857 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:44.529 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.887 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:44.560 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.932 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:44.579 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.937 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:44.583 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.951 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:44.624 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.956 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:44.603 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.958 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:44.630 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:39.983 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:44.655 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.623 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:46.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.740+0000 7fa09c64e640 10 monclient: get_monmap_and_config 2024-03-25T19:30:46.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.740+0000 7fa09c64e640 10 monclient: build_initial_monmap 2024-03-25T19:30:46.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.740+0000 7fa09c64e640 10 monclient: monmap: 2024-03-25T19:30:46.740 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:46.740 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:46.741591+0000 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:46.741591+0000 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:46.741 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:46.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.741+0000 7fa09c64e640 1 Processor -- start 2024-03-25T19:30:46.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.741+0000 7fa09c64e640 1 -- start start 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.741+0000 7fa09c64e640 10 monclient: init 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 10 monclient(hunting): picked mon.noname-c con 0x7fa0940a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 10 monclient(hunting): picked mon.noname-b con 0x7fa09413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 10 monclient(hunting): picked mon.noname-a con 0x7fa0941430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa094142520 con 0x7fa0941430c0 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0940b28a0 con 0x7fa0940a98f0 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0940b2a30 con 0x7fa09413f710 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:46.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.742+0000 7fa09c64e640 10 monclient(hunting): authenticate will time out at 6078.900953s 2024-03-25T19:30:46.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.747+0000 7fa09abeb640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa0941430c0 0x7fa09414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:38734/0 (socket says 172.21.15.134:38734) 2024-03-25T19:30:46.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.747+0000 7fa09abeb640 1 -- 172.21.15.134:0/847247958 learned_addr learned my addr 172.21.15.134:0/847247958 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596888223 0 0) 0x7fa0940b2a30 con 0x7fa09413f710 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): _negotiate 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa080003520 con 0x7fa09413f710 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321521074 0 0) 0x7fa094142520 con 0x7fa0941430c0 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): _negotiate 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:30:46.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.760+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0940b2a30 con 0x7fa0941430c0 2024-03-25T19:30:46.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3592881452 0 0) 0x7fa080003520 con 0x7fa09413f710 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 >> v1:172.21.15.159:6789/0 conn(0x7fa0940a98f0 legacy=0x7fa0940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 >> v1:172.21.15.134:6789/0 conn(0x7fa0941430c0 legacy=0x7fa09414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 10 monclient: _finish_hunting 0 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 1 monclient: found mon.noname-b 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940b44c0 con 0x7fa09413f710 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 10 monclient: _finish_auth 0 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 10 monclient: _check_auth_tickets 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa084001690 con 0x7fa09413f710 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa0993e8640 10 monclient: dump: 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:46.771 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa09c64e640 5 monclient: authenticate success, global_id 4319 2024-03-25T19:30:46.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.770+0000 7fa09c64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0840017f0 con 0x7fa09413f710 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa0993e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa0993e8640 1 -- 172.21.15.134:0/847247958 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa084001c00 con 0x7fa09413f710 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa09c64e640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa09c64e640 4 set_mon_vals no callback set 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa0993e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa0993e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.796+0000 7fa0993e8640 10 monclient: dump: 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:46.796 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.797 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.797 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:46.797 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:46.797 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:46.797 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:46.797 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:46.797 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:46.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 10 monclient: shutdown 2024-03-25T19:30:46.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:46.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 1 -- 172.21.15.134:0/847247958 >> v1:172.21.15.134:6790/0 conn(0x7fa09413f710 legacy=0x7fa0941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:46.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 1 -- 172.21.15.134:0/847247958 shutdown_connections 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 1 -- 172.21.15.134:0/847247958 >> 172.21.15.134:0/847247958 conn(0x7fa0940b7ea0 msgr2=0x7fa0940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 1 -- 172.21.15.134:0/847247958 shutdown_connections 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 1 -- 172.21.15.134:0/847247958 wait complete. 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 10 monclient: build_initial_monmap 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.797+0000 7fa09c64e640 10 monclient: monmap: 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.798+0000 7fa09c64e640 1 librados: starting msgr at 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.798+0000 7fa09c64e640 1 librados: starting objecter 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.798+0000 7fa09c64e640 1 Processor -- start 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.798+0000 7fa09c64e640 1 -- start start 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.798+0000 7fa09c64e640 1 librados: setting wanted keys 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.798+0000 7fa09c64e640 1 librados: calling monclient init 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.798+0000 7fa09c64e640 10 monclient: init 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 10 monclient(hunting): picked mon.noname-a con 0x7fa0940a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 10 monclient(hunting): picked mon.noname-b con 0x7fa09413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 10 monclient(hunting): picked mon.noname-c con 0x7fa0941430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0940abc80 con 0x7fa0940a98f0 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0941d7780 con 0x7fa09413f710 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0941399a0 con 0x7fa0941430c0 2024-03-25T19:30:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:46.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.799+0000 7fa09c64e640 10 monclient(hunting): authenticate will time out at 6078.958245s 2024-03-25T19:30:46.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.805+0000 7fa09a3ea640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa0940a98f0 0x7fa0941d7c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:38792/0 (socket says 172.21.15.134:38792) 2024-03-25T19:30:46.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.805+0000 7fa09a3ea640 1 -- 172.21.15.134:0/989570763 learned_addr learned my addr 172.21.15.134:0/989570763 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.807+0000 7fa08affd640 10 client.?.objecter ms_handle_connect 0x7fa0940a98f0 2024-03-25T19:30:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.807+0000 7fa08affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:46.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.812+0000 7fa08affd640 10 client.?.objecter ms_handle_connect 0x7fa0941430c0 2024-03-25T19:30:46.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.812+0000 7fa08affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:46.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.816+0000 7fa08affd640 10 client.?.objecter ms_handle_connect 0x7fa09413f710 2024-03-25T19:30:46.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.816+0000 7fa08affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:46.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.819+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761016569 0 0) 0x7fa0941399a0 con 0x7fa0941430c0 2024-03-25T19:30:46.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.819+0000 7fa08affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:46.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.819+0000 7fa08affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:46.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.819+0000 7fa08affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:46.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.819+0000 7fa08affd640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:30:46.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.819+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa078004eb0 con 0x7fa0941430c0 2024-03-25T19:30:46.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.820+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79746189 0 0) 0x7fa0941d7780 con 0x7fa09413f710 2024-03-25T19:30:46.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.820+0000 7fa08affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:46.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.820+0000 7fa08affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:46.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.820+0000 7fa08affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:46.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.820+0000 7fa08affd640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:30:46.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.820+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0941399a0 con 0x7fa09413f710 2024-03-25T19:30:46.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.822+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067683835 0 0) 0x7fa0940abc80 con 0x7fa0940a98f0 2024-03-25T19:30:46.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.822+0000 7fa08affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:46.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.822+0000 7fa08affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:46.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.822+0000 7fa08affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:46.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.822+0000 7fa08affd640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:30:46.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.822+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0941d7780 con 0x7fa0940a98f0 2024-03-25T19:30:46.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.834+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 997117047 0 0) 0x7fa0941d7780 con 0x7fa0940a98f0 2024-03-25T19:30:46.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.834+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0940abc80 con 0x7fa0940a98f0 2024-03-25T19:30:46.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.834+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa090001970 con 0x7fa0940a98f0 2024-03-25T19:30:46.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.834+0000 7fa08affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:46.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.834+0000 7fa08affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:46.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.834+0000 7fa08affd640 10 monclient(hunting): dump: 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:46.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.834+0000 7fa08affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:46.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.835+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 587057765 0 0) 0x7fa078004eb0 con 0x7fa0941430c0 2024-03-25T19:30:46.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.835+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0941d7780 con 0x7fa0941430c0 2024-03-25T19:30:46.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.835+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa08c001bd0 con 0x7fa0941430c0 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.835+0000 7fa08affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.835+0000 7fa08affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.835+0000 7fa08affd640 10 monclient(hunting): dump: 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.836+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 496247394 0 0) 0x7fa0941399a0 con 0x7fa09413f710 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.836+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa078004eb0 con 0x7fa09413f710 2024-03-25T19:30:46.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.836+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa084001b00 con 0x7fa09413f710 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.836+0000 7fa08affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.836+0000 7fa08affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.836+0000 7fa08affd640 10 monclient(hunting): dump: 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:46.838 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 177258832 0 0) 0x7fa078004eb0 con 0x7fa09413f710 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 >> v1:172.21.15.134:6790/0 conn(0x7fa0941430c0 legacy=0x7fa094138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 >> v1:172.21.15.134:6789/0 conn(0x7fa0940a98f0 legacy=0x7fa0941d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 1 monclient: found mon.b 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0941d41e0 con 0x7fa09413f710 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 10 monclient: _finish_auth 0 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:46.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa08affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa09c64e640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa09c64e640 10 monclient: _renew_subs 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa09c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.845+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0941d8370 con 0x7fa09413f710 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 10 monclient: _renew_subs 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0941d5130 con 0x7fa09413f710 2024-03-25T19:30:46.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 1 librados: init done 2024-03-25T19:30:46.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:46.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:46.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:46.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:46.846+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0940627b0 con 0x7fa09413f710 2024-03-25T19:30:47.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.139+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0840038a0 con 0x7fa09413f710 2024-03-25T19:30:47.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.139+0000 7fa08affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:47.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.139+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa084003bb0 con 0x7fa09413f710 2024-03-25T19:30:47.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.139+0000 7fa08affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:47.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.139+0000 7fa08affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:47.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.139+0000 7fa08affd640 10 monclient: dump: 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:47.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.139+0000 7fa098be7640 4 set_mon_vals no callback set 2024-03-25T19:30:47.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.140+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fa084019b80 con 0x7fa09413f710 2024-03-25T19:30:47.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.141+0000 7fa09a3ea640 10 monclient: get_auth_request con 0x7fa07806e160 auth_method 0 2024-03-25T19:30:47.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.141+0000 7fa08affd640 10 client.4321.objecter ms_handle_connect 0x7fa07806e160 2024-03-25T19:30:47.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.145+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fa0840535c0 con 0x7fa09413f710 2024-03-25T19:30:47.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.145+0000 7fa08affd640 10 client.4321.objecter ms_dispatch 0x7fa094000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:47.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.145+0000 7fa08affd640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:47.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.145+0000 7fa08affd640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:47.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.145+0000 7fa08affd640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:30:47.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.146+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa08404f0d0 con 0x7fa09413f710 2024-03-25T19:30:47.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.146+0000 7fa08affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:47.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.146+0000 7fa08affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:47.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.385+0000 7fa09c64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:47.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.385+0000 7fa09c64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:47.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.385+0000 7fa09c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:30:47.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.385+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa09413e2b0 con 0x7fa09413f710 2024-03-25T19:30:47.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.418+0000 7fa08affd640 1 -- 172.21.15.134:0/989570763 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fa084051000 con 0x7fa09413f710 2024-03-25T19:30:47.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.418+0000 7fa08affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:47.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.418+0000 7fa08affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:47.418 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:47.418 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 10 librados: watch_flush enter 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 10 librados: watch_flush exit 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 >> v1:172.21.15.159:6800/1875874887 conn(0x7fa07806e160 legacy=0x7fa078070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 10 monclient: shutdown 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 >> v1:172.21.15.159:6789/0 conn(0x7fa09413f710 legacy=0x7fa0941d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 shutdown_connections 2024-03-25T19:30:47.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 >> 172.21.15.134:0/989570763 conn(0x7fa0940b7ea0 msgr2=0x7fa094074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:47.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 shutdown_connections 2024-03-25T19:30:47.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 1 -- 172.21.15.134:0/989570763 wait complete. 2024-03-25T19:30:47.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:47.421+0000 7fa09c64e640 1 librados: shutdown 2024-03-25T19:30:47.434 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:54.241 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:54.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.358+0000 7f6d69932640 10 monclient: get_monmap_and_config 2024-03-25T19:30:54.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.358+0000 7f6d69932640 10 monclient: build_initial_monmap 2024-03-25T19:30:54.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.358+0000 7f6d69932640 10 monclient: monmap: 2024-03-25T19:30:54.358 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:30:54.359798+0000 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:30:54.359798+0000 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:30:54.359 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.359+0000 7f6d69932640 1 Processor -- start 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.359+0000 7f6d69932640 1 -- start start 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.359+0000 7f6d69932640 10 monclient: init 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 10 monclient(hunting): picked mon.noname-a con 0x7f6d640a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 10 monclient(hunting): picked mon.noname-c con 0x7f6d6413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 10 monclient(hunting): picked mon.noname-b con 0x7f6d641430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d64142520 con 0x7f6d640a98f0 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d640b28a0 con 0x7f6d6413f710 2024-03-25T19:30:54.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d640b2a30 con 0x7f6d641430c0 2024-03-25T19:30:54.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:54.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d69932640 10 monclient(hunting): authenticate will time out at 6086.519030s 2024-03-25T19:30:54.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d627fc640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f6d6413f710 0x7f6d641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:42738/0 (socket says 172.21.15.134:42738) 2024-03-25T19:30:54.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.360+0000 7f6d627fc640 1 -- 172.21.15.134:0/434244648 learned_addr learned my addr 172.21.15.134:0/434244648 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:54.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.444+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776134409 0 0) 0x7f6d64142520 con 0x7f6d640a98f0 2024-03-25T19:30:54.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.444+0000 7f6d61ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:54.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.444+0000 7f6d61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:54.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.444+0000 7f6d61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:54.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.444+0000 7f6d61ffb640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:30:54.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.444+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d4c003520 con 0x7f6d640a98f0 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1576756401 0 0) 0x7f6d4c003520 con 0x7f6d640a98f0 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 >> v1:172.21.15.134:6790/0 conn(0x7f6d641430c0 legacy=0x7f6d6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 >> v1:172.21.15.159:6789/0 conn(0x7f6d6413f710 legacy=0x7f6d641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 1 monclient: found mon.noname-a 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:30:54.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d640b44c0 con 0x7f6d640a98f0 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6d54001690 con 0x7f6d640a98f0 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d61ffb640 10 monclient: dump: 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:54.458 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:54.459 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:54.459 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:54.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d69932640 5 monclient: authenticate success, global_id 4350 2024-03-25T19:30:54.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.457+0000 7f6d69932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:54.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d540017f0 con 0x7f6d640a98f0 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d61ffb640 1 -- 172.21.15.134:0/434244648 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6d54001c00 con 0x7f6d640a98f0 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d69932640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d69932640 4 set_mon_vals no callback set 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d61ffb640 10 monclient: dump: 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:54.472 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:54.473 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:54.473 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:54.473 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:54.473 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:54.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d69932640 10 monclient: shutdown 2024-03-25T19:30:54.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d69932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:54.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.472+0000 7f6d69932640 1 -- 172.21.15.134:0/434244648 >> v1:172.21.15.134:6789/0 conn(0x7f6d640a98f0 legacy=0x7f6d640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:54.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.473+0000 7f6d69932640 1 -- 172.21.15.134:0/434244648 shutdown_connections 2024-03-25T19:30:54.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.473+0000 7f6d69932640 1 -- 172.21.15.134:0/434244648 >> 172.21.15.134:0/434244648 conn(0x7f6d640b7ea0 msgr2=0x7f6d640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:54.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.473+0000 7f6d69932640 1 -- 172.21.15.134:0/434244648 shutdown_connections 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.473+0000 7f6d69932640 1 -- 172.21.15.134:0/434244648 wait complete. 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 10 monclient: build_initial_monmap 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 10 monclient: monmap: 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:30:54.475 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:54.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 1 librados: starting msgr at 2024-03-25T19:30:54.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 1 librados: starting objecter 2024-03-25T19:30:54.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 1 Processor -- start 2024-03-25T19:30:54.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 1 -- start start 2024-03-25T19:30:54.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 1 librados: setting wanted keys 2024-03-25T19:30:54.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 1 librados: calling monclient init 2024-03-25T19:30:54.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.474+0000 7f6d69932640 10 monclient: init 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 10 monclient(hunting): picked mon.noname-c con 0x7f6d640a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 10 monclient(hunting): picked mon.noname-a con 0x7f6d6413f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 10 monclient(hunting): picked mon.noname-b con 0x7f6d641430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d64138c70 con 0x7f6d6413f710 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d641367d0 con 0x7f6d641430c0 2024-03-25T19:30:54.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d64136960 con 0x7f6d640a98f0 2024-03-25T19:30:54.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:54.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.475+0000 7f6d69932640 10 monclient(hunting): authenticate will time out at 6086.634519s 2024-03-25T19:30:54.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.497+0000 7f6d627fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6d6413f710 0x7f6d64132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58088/0 (socket says 172.21.15.134:58088) 2024-03-25T19:30:54.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.497+0000 7f6d627fc640 1 -- 172.21.15.134:0/2927150985 learned_addr learned my addr 172.21.15.134:0/2927150985 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:30:54.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.498+0000 7f6d4bfff640 10 client.?.objecter ms_handle_connect 0x7f6d6413f710 2024-03-25T19:30:54.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.498+0000 7f6d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:54.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322665934 0 0) 0x7f6d64138c70 con 0x7f6d6413f710 2024-03-25T19:30:54.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:54.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:54.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:54.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:30:54.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d50004eb0 con 0x7f6d6413f710 2024-03-25T19:30:54.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 10 client.?.objecter ms_handle_connect 0x7f6d641430c0 2024-03-25T19:30:54.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.506+0000 7f6d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:54.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.515+0000 7f6d4bfff640 10 client.?.objecter ms_handle_connect 0x7f6d640a98f0 2024-03-25T19:30:54.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.515+0000 7f6d4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:54.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.519+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641932775 0 0) 0x7f6d50004eb0 con 0x7f6d6413f710 2024-03-25T19:30:54.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.519+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d64138c70 con 0x7f6d6413f710 2024-03-25T19:30:54.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.521+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6d58001990 con 0x7f6d6413f710 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.521+0000 7f6d4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.521+0000 7f6d4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.521+0000 7f6d4bfff640 10 monclient(hunting): dump: 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:54.521 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:54.522 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:54.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.521+0000 7f6d4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:54.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1176369952 0 0) 0x7f6d64138c70 con 0x7f6d6413f710 2024-03-25T19:30:54.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 >> v1:172.21.15.134:6790/0 conn(0x7f6d640a98f0 legacy=0x7f6d64137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:54.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 >> v1:172.21.15.159:6789/0 conn(0x7f6d641430c0 legacy=0x7f6d641e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 1 monclient: found mon.a 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d64139630 con 0x7f6d6413f710 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 10 monclient: _finish_auth 0 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 10 monclient: _renew_subs 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d64139880 con 0x7f6d6413f710 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:54.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 10 monclient: _renew_subs 2024-03-25T19:30:54.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:54.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d641337f0 con 0x7f6d6413f710 2024-03-25T19:30:54.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.529+0000 7f6d69932640 1 librados: init done 2024-03-25T19:30:54.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.530+0000 7f6d69932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:54.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.530+0000 7f6d69932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:54.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.530+0000 7f6d69932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:54.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.530+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d640627b0 con 0x7f6d6413f710 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.827+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d580047e0 con 0x7f6d6413f710 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.827+0000 7f6d4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.827+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6d58004c30 con 0x7f6d6413f710 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.827+0000 7f6d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.827+0000 7f6d4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.827+0000 7f6d4bfff640 10 monclient: dump: 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.828 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.827+0000 7f6d617fa640 4 set_mon_vals no callback set 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.828+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f6d58004e50 con 0x7f6d6413f710 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.829+0000 7f6d62ffd640 10 monclient: get_auth_request con 0x7f6d5006dcb0 auth_method 0 2024-03-25T19:30:54.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.829+0000 7f6d4bfff640 10 client.4353.objecter ms_handle_connect 0x7f6d5006dcb0 2024-03-25T19:30:54.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f6d580548e0 con 0x7f6d6413f710 2024-03-25T19:30:54.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 10 client.4353.objecter ms_dispatch 0x7f6d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:54.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:54.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:54.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T19:30:54.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d580503f0 con 0x7f6d6413f710 2024-03-25T19:30:54.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:54.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:54.835+0000 7f6d4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:55.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.094+0000 7f6d69932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:55.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.094+0000 7f6d69932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:55.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.094+0000 7f6d69932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:30:55.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.094+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d64137370 con 0x7f6d6413f710 2024-03-25T19:30:55.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.122+0000 7f6d4bfff640 1 -- 172.21.15.134:0/2927150985 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f6d58052320 con 0x7f6d6413f710 2024-03-25T19:30:55.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.122+0000 7f6d4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:55.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.122+0000 7f6d4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:55.122 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:30:55.123 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:55.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 10 librados: watch_flush enter 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 10 librados: watch_flush exit 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 10 client.4353.objecter successfully canceled tick 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 >> v1:172.21.15.159:6800/1875874887 conn(0x7f6d5006dcb0 legacy=0x7f6d50070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 10 monclient: shutdown 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 >> v1:172.21.15.134:6789/0 conn(0x7f6d6413f710 legacy=0x7f6d64132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 shutdown_connections 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 >> 172.21.15.134:0/2927150985 conn(0x7f6d640b7ea0 msgr2=0x7f6d64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 shutdown_connections 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 1 -- 172.21.15.134:0/2927150985 wait complete. 2024-03-25T19:30:55.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:30:55.125+0000 7f6d69932640 1 librados: shutdown 2024-03-25T19:30:55.137 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:59.927 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:04.599 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:59.945 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:04.617 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:59.983 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:04.655 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:59.986 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:04.633 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:59.993 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:04.640 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:59.994 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:04.666 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:00.001 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:04.674 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:00.013 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:04.660 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:00.035 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:04.708 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.943 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:02.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.061+0000 7fb4ceab3640 10 monclient: get_monmap_and_config 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.061+0000 7fb4ceab3640 10 monclient: build_initial_monmap 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.061+0000 7fb4ceab3640 10 monclient: monmap: 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:02.062246+0000 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:02.062246+0000 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:02.061 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:02.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.062+0000 7fb4ceab3640 1 Processor -- start 2024-03-25T19:31:02.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.062+0000 7fb4ceab3640 1 -- start start 2024-03-25T19:31:02.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.062+0000 7fb4ceab3640 10 monclient: init 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 10 monclient(hunting): picked mon.noname-b con 0x7fb4c80705b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 10 monclient(hunting): picked mon.noname-c con 0x7fb4c81402d0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 10 monclient(hunting): picked mon.noname-a con 0x7fb4c8149ae0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80718c0 con 0x7fb4c8149ae0 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80714c0 con 0x7fb4c81402d0 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c8072b50 con 0x7fb4c80705b0 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.063+0000 7fb4ceab3640 10 monclient(hunting): authenticate will time out at 6094.221784s 2024-03-25T19:31:02.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.071+0000 7fb4cc84f640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fb4c80705b0 0x7fb4c813d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33842/0 (socket says 172.21.15.134:33842) 2024-03-25T19:31:02.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.071+0000 7fb4cc84f640 1 -- 172.21.15.134:0/4208401787 learned_addr learned my addr 172.21.15.134:0/4208401787 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:02.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.081+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668587616 0 0) 0x7fb4c8072b50 con 0x7fb4c80705b0 2024-03-25T19:31:02.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.081+0000 7fb4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:02.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.081+0000 7fb4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:02.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.081+0000 7fb4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:02.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.081+0000 7fb4c77fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:31:02.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.081+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b8003520 con 0x7fb4c80705b0 2024-03-25T19:31:02.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.084+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875494911 0 0) 0x7fb4c80714c0 con 0x7fb4c81402d0 2024-03-25T19:31:02.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.085+0000 7fb4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:02.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.085+0000 7fb4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:02.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.085+0000 7fb4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:02.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.085+0000 7fb4c77fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:31:02.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.085+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c8072b50 con 0x7fb4c81402d0 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2824089504 0 0) 0x7fb4b8003520 con 0x7fb4c80705b0 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 >> v1:172.21.15.159:6789/0 conn(0x7fb4c81402d0 legacy=0x7fb4c8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 >> v1:172.21.15.134:6789/0 conn(0x7fb4c8149ae0 legacy=0x7fb4c814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 1 monclient: found mon.noname-b 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c813dd10 con 0x7fb4c80705b0 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:02.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb4bc001690 con 0x7fb4c80705b0 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4ceab3640 5 monclient: authenticate success, global_id 4328 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4ceab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4c77fe640 10 monclient: dump: 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:02.096 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:02.097 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:02.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.094+0000 7fb4ceab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4bc0017f0 con 0x7fb4c80705b0 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4c77fe640 1 -- 172.21.15.134:0/4208401787 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb4bc001c00 con 0x7fb4c80705b0 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4ceab3640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4ceab3640 4 set_mon_vals no callback set 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:02.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.112+0000 7fb4c77fe640 10 monclient: dump: 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:02.113 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:02.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.113+0000 7fb4ceab3640 10 monclient: shutdown 2024-03-25T19:31:02.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.113+0000 7fb4ceab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:02.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.113+0000 7fb4ceab3640 1 -- 172.21.15.134:0/4208401787 >> v1:172.21.15.134:6790/0 conn(0x7fb4c80705b0 legacy=0x7fb4c813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:02.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.113+0000 7fb4ceab3640 1 -- 172.21.15.134:0/4208401787 shutdown_connections 2024-03-25T19:31:02.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.113+0000 7fb4ceab3640 1 -- 172.21.15.134:0/4208401787 >> 172.21.15.134:0/4208401787 conn(0x7fb4c80b5d10 msgr2=0x7fb4c80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:02.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.113+0000 7fb4ceab3640 1 -- 172.21.15.134:0/4208401787 shutdown_connections 2024-03-25T19:31:02.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.113+0000 7fb4ceab3640 1 -- 172.21.15.134:0/4208401787 wait complete. 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 10 monclient: build_initial_monmap 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 10 monclient: monmap: 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 1 librados: starting msgr at 2024-03-25T19:31:02.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 1 librados: starting objecter 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 1 Processor -- start 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 1 -- start start 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 1 librados: setting wanted keys 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 1 librados: calling monclient init 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.114+0000 7fb4ceab3640 10 monclient: init 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 10 monclient(hunting): picked mon.noname-c con 0x7fb4c80705b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 10 monclient(hunting): picked mon.noname-a con 0x7fb4c81402d0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 10 monclient(hunting): picked mon.noname-b con 0x7fb4c8149ae0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c8147c30 con 0x7fb4c81402d0 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c8130660 con 0x7fb4c8149ae0 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c81455d0 con 0x7fb4c80705b0 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:02.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.115+0000 7fb4ceab3640 10 monclient(hunting): authenticate will time out at 6094.274403s 2024-03-25T19:31:02.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.122+0000 7fb4cd050640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb4c8149ae0 0x7fb4c8148940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:60938/0 (socket says 172.21.15.134:60938) 2024-03-25T19:31:02.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.122+0000 7fb4cd050640 1 -- 172.21.15.134:0/3875432192 learned_addr learned my addr 172.21.15.134:0/3875432192 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:02.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.124+0000 7fb4c57fa640 10 client.?.objecter ms_handle_connect 0x7fb4c8149ae0 2024-03-25T19:31:02.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.124+0000 7fb4c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:02.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.127+0000 7fb4cd050640 1 -- 172.21.15.134:0/3875432192 >> v1:172.21.15.159:6789/0 conn(0x7fb4c8149ae0 legacy=0x7fb4c8148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:31:02.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.127+0000 7fb4cd050640 1 -- 172.21.15.134:0/3875432192 >> v1:172.21.15.159:6789/0 conn(0x7fb4c8149ae0 legacy=0x7fb4c8148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:31:02.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.127+0000 7fb4cd050640 1 --1- 172.21.15.134:0/3875432192 >> v1:172.21.15.159:6789/0 conn(0x7fb4c8149ae0 0x7fb4c8148940 :-1 s=OPENED pgs=15706 cs=1 l=1).handle_message read tag failed 2024-03-25T19:31:02.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.127+0000 7fb4cd050640 1 --1- 172.21.15.134:0/3875432192 >> v1:172.21.15.159:6789/0 conn(0x7fb4c8149ae0 0x7fb4c8148940 :-1 s=OPENED pgs=15706 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:31:02.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.127+0000 7fb4c57fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.159:6789/0 2024-03-25T19:31:02.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.129+0000 7fb4c57fa640 10 client.?.objecter ms_handle_connect 0x7fb4c80705b0 2024-03-25T19:31:02.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.129+0000 7fb4c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:02.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.132+0000 7fb4c57fa640 10 client.?.objecter ms_handle_connect 0x7fb4c81402d0 2024-03-25T19:31:02.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.132+0000 7fb4c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:02.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.135+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140687430 0 0) 0x7fb4c81455d0 con 0x7fb4c80705b0 2024-03-25T19:31:02.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.135+0000 7fb4c57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:02.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.135+0000 7fb4c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:02.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.135+0000 7fb4c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:02.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.135+0000 7fb4c57fa640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:31:02.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.135+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ac004eb0 con 0x7fb4c80705b0 2024-03-25T19:31:02.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.140+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250181854 0 0) 0x7fb4c8147c30 con 0x7fb4c81402d0 2024-03-25T19:31:02.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.140+0000 7fb4c57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:02.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.140+0000 7fb4c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:02.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.140+0000 7fb4c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:02.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.140+0000 7fb4c57fa640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:31:02.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.140+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c81455d0 con 0x7fb4c81402d0 2024-03-25T19:31:02.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.147+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885251934 0 0) 0x7fb4ac004eb0 con 0x7fb4c80705b0 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.147+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4c8147c30 con 0x7fb4c80705b0 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.147+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb4bc001bc0 con 0x7fb4c80705b0 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.147+0000 7fb4c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.147+0000 7fb4c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.147+0000 7fb4c57fa640 10 monclient(hunting): dump: 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:02.148 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:02.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.148+0000 7fb4c57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.152+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2404506159 0 0) 0x7fb4c81455d0 con 0x7fb4c81402d0 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.152+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4ac004eb0 con 0x7fb4c81402d0 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.152+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb4b4001930 con 0x7fb4c81402d0 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.152+0000 7fb4c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.152+0000 7fb4c57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.152+0000 7fb4c57fa640 10 monclient(hunting): dump: 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:02.152 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:02.153 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:02.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2395229531 0 0) 0x7fb4c8147c30 con 0x7fb4c80705b0 2024-03-25T19:31:02.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 >> v1:172.21.15.159:6789/0 conn(0x7fb4c8149ae0 legacy=0x7fb4c8148940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:31:02.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 >> v1:172.21.15.134:6789/0 conn(0x7fb4c81402d0 legacy=0x7fb4c8148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:02.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 1 monclient: found mon.c 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c8145760 con 0x7fb4c80705b0 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 10 monclient: _finish_auth 0 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 5 monclient: authenticate success, global_id 4331 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 10 monclient: _renew_subs 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c8146780 con 0x7fb4c80705b0 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:02.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 10 monclient: _renew_subs 2024-03-25T19:31:02.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:02.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c81e37b0 con 0x7fb4c80705b0 2024-03-25T19:31:02.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.172+0000 7fb4ceab3640 1 librados: init done 2024-03-25T19:31:02.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.173+0000 7fb4ceab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:02.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.173+0000 7fb4ceab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:02.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.173+0000 7fb4ceab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:02.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.173+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c80627b0 con 0x7fb4c80705b0 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.475+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4bc004a90 con 0x7fb4c80705b0 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.475+0000 7fb4c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.475+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb4bc004da0 con 0x7fb4c80705b0 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.475+0000 7fb4c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.475+0000 7fb4c57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.475+0000 7fb4c57fa640 10 monclient: dump: 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:02.475 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:02.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.475+0000 7fb4c6ffd640 4 set_mon_vals no callback set 2024-03-25T19:31:02.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.476+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fb4bc01ad70 con 0x7fb4c80705b0 2024-03-25T19:31:02.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.476+0000 7fb4c7fff640 10 monclient: get_auth_request con 0x7fb4ac06de90 auth_method 0 2024-03-25T19:31:02.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.477+0000 7fb4c57fa640 10 client.4331.objecter ms_handle_connect 0x7fb4ac06de90 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fb4bc054930 con 0x7fb4c80705b0 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 10 client.4331.objecter ms_dispatch 0x7fb4c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4bc050440 con 0x7fb4c80705b0 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:02.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.481+0000 7fb4c57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:02.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.701+0000 7fb4ceab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:02.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.701+0000 7fb4ceab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:02.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.701+0000 7fb4ceab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:02.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.701+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c8150400 con 0x7fb4c80705b0 2024-03-25T19:31:02.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.769+0000 7fb4c57fa640 1 -- 172.21.15.134:0/3875432192 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fb4bc052370 con 0x7fb4c80705b0 2024-03-25T19:31:02.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.769+0000 7fb4c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:02.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.769+0000 7fb4c57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:02.770 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:02.770 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:02.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 10 librados: watch_flush enter 2024-03-25T19:31:02.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 10 librados: watch_flush exit 2024-03-25T19:31:02.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-25T19:31:02.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 10 client.4331.objecter successfully canceled tick 2024-03-25T19:31:02.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 >> v1:172.21.15.159:6800/1875874887 conn(0x7fb4ac06de90 legacy=0x7fb4ac070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:02.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 10 monclient: shutdown 2024-03-25T19:31:02.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:02.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 >> v1:172.21.15.134:6790/0 conn(0x7fb4c80705b0 legacy=0x7fb4c8146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:02.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 shutdown_connections 2024-03-25T19:31:02.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 >> 172.21.15.134:0/3875432192 conn(0x7fb4c80b5d10 msgr2=0x7fb4c80b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:02.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 shutdown_connections 2024-03-25T19:31:02.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 1 -- 172.21.15.134:0/3875432192 wait complete. 2024-03-25T19:31:02.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:02.772+0000 7fb4ceab3640 1 librados: shutdown 2024-03-25T19:31:02.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:09.590 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:09.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.710+0000 7f8bd35a3640 10 monclient: get_monmap_and_config 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.710+0000 7f8bd35a3640 10 monclient: build_initial_monmap 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.710+0000 7f8bd35a3640 10 monclient: monmap: 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:09.711834+0000 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:09.711834+0000 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:09.711 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:09.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.711+0000 7f8bd35a3640 1 Processor -- start 2024-03-25T19:31:09.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.711+0000 7f8bd35a3640 1 -- start start 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.711+0000 7f8bd35a3640 10 monclient: init 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 10 monclient(hunting): picked mon.noname-b con 0x7f8bcc0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 10 monclient(hunting): picked mon.noname-a con 0x7f8bcc13f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc142520 con 0x7f8bcc13f710 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0b28a0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0b2a30 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:09.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.712+0000 7f8bd35a3640 10 monclient(hunting): authenticate will time out at 6101.871413s 2024-03-25T19:31:09.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.720+0000 7f8bd133f640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f8bcc0a98f0 0x7f8bcc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:38558/0 (socket says 172.21.15.134:38558) 2024-03-25T19:31:09.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.720+0000 7f8bd133f640 1 -- 172.21.15.134:0/2253613284 learned_addr learned my addr 172.21.15.134:0/2253613284 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:09.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.733+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461065581 0 0) 0x7f8bcc0b2a30 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.733+0000 7f8bc3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:09.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.733+0000 7f8bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:09.733 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.733+0000 7f8bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:09.733 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.733+0000 7f8bc3fff640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:31:09.733 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.733+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8003520 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678414817 0 0) 0x7f8bcc0b28a0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:09.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:09.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:09.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:31:09.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc0b2a30 con 0x7f8bcc1430c0 2024-03-25T19:31:09.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720699187 0 0) 0x7f8bcc142520 con 0x7f8bcc13f710 2024-03-25T19:31:09.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:09.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:09.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:09.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:31:09.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.734+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc0b28a0 con 0x7f8bcc13f710 2024-03-25T19:31:09.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2118501309 0 0) 0x7f8bcc0b2a30 con 0x7f8bcc1430c0 2024-03-25T19:31:09.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 >> v1:172.21.15.134:6790/0 conn(0x7f8bcc0a98f0 legacy=0x7f8bcc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 >> v1:172.21.15.134:6789/0 conn(0x7f8bcc13f710 legacy=0x7f8bcc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 1 monclient: found mon.noname-c 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc0b44c0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8bc40019f0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bd35a3640 5 monclient: authenticate success, global_id 4330 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bd35a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:09.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bc3fff640 10 monclient: dump: 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:09.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.746+0000 7f8bd35a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc4001b50 con 0x7f8bcc1430c0 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bc3fff640 1 -- 172.21.15.134:0/2253613284 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8bc4001f60 con 0x7f8bcc1430c0 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bd35a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bd35a3640 4 set_mon_vals no callback set 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bc3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.758+0000 7f8bc3fff640 10 monclient: dump: 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:09.759 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:09.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.759+0000 7f8bd35a3640 10 monclient: shutdown 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.759+0000 7f8bd35a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.759+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2253613284 >> v1:172.21.15.159:6789/0 conn(0x7f8bcc1430c0 legacy=0x7f8bcc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.759+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2253613284 shutdown_connections 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.759+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2253613284 >> 172.21.15.134:0/2253613284 conn(0x7f8bcc0b7ea0 msgr2=0x7f8bcc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.759+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2253613284 shutdown_connections 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.759+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2253613284 wait complete. 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 10 monclient: build_initial_monmap 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 10 monclient: monmap: 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:09.761 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 1 librados: starting msgr at 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 1 librados: starting objecter 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 1 Processor -- start 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 1 -- start start 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 1 librados: setting wanted keys 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 1 librados: calling monclient init 2024-03-25T19:31:09.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.760+0000 7f8bd35a3640 10 monclient: init 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 10 monclient(hunting): picked mon.noname-a con 0x7f8bcc0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc13f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 10 monclient(hunting): picked mon.noname-b con 0x7f8bcc1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc13c190 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0b7cf0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc13ad50 con 0x7f8bcc13f710 2024-03-25T19:31:09.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:09.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.761+0000 7f8bd35a3640 10 monclient(hunting): authenticate will time out at 6101.920454s 2024-03-25T19:31:09.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.769+0000 7f8bd133f640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8bcc0a98f0 0x7f8bcc13ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:50060/0 (socket says 172.21.15.134:50060) 2024-03-25T19:31:09.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.769+0000 7f8bd133f640 1 -- 172.21.15.134:0/2776719884 learned_addr learned my addr 172.21.15.134:0/2776719884 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:09.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.771+0000 7f8bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f8bcc0a98f0 2024-03-25T19:31:09.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.771+0000 7f8bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:09.774 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.774+0000 7f8bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f8bcc13f710 2024-03-25T19:31:09.774 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.774+0000 7f8bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:09.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.779+0000 7f8bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f8bcc1430c0 2024-03-25T19:31:09.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.779+0000 7f8bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:09.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.783+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135136587 0 0) 0x7f8bb8003520 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.783+0000 7f8bc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:09.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.783+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:09.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.783+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:09.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.783+0000 7f8bc1ffb640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:31:09.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.783+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0004eb0 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.793+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318092578 0 0) 0x7f8bcc13ad50 con 0x7f8bcc13f710 2024-03-25T19:31:09.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.793+0000 7f8bc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:09.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.793+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:09.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.793+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:09.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.793+0000 7f8bc1ffb640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:31:09.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.793+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8003520 con 0x7f8bcc13f710 2024-03-25T19:31:09.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.795+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087633601 0 0) 0x7f8bcc0b7cf0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.795+0000 7f8bc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:09.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.795+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:09.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.795+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:09.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.795+0000 7f8bc1ffb640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:31:09.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.795+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc13ad50 con 0x7f8bcc1430c0 2024-03-25T19:31:09.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.807+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346319640 0 0) 0x7f8bb8003520 con 0x7f8bcc13f710 2024-03-25T19:31:09.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.807+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bcc0b7cf0 con 0x7f8bcc13f710 2024-03-25T19:31:09.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.807+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8bbc001870 con 0x7f8bcc13f710 2024-03-25T19:31:09.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.807+0000 7f8bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:09.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.807+0000 7f8bc1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:09.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.807+0000 7f8bc1ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:09.807 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:09.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.807+0000 7f8bc1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:09.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.810+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1078084874 0 0) 0x7f8bcc13ad50 con 0x7f8bcc1430c0 2024-03-25T19:31:09.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.810+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bb8003520 con 0x7f8bcc1430c0 2024-03-25T19:31:09.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.810+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8bc4001e60 con 0x7f8bcc1430c0 2024-03-25T19:31:09.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.810+0000 7f8bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:09.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.810+0000 7f8bc1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:09.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.810+0000 7f8bc1ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:09.810 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:09.811 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:09.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.814+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2949065999 0 0) 0x7f8bb0004eb0 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.814+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bcc13ad50 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.814+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8bc80017b0 con 0x7f8bcc0a98f0 2024-03-25T19:31:09.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.814+0000 7f8bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:09.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.814+0000 7f8bc1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.814+0000 7f8bc1ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:09.815 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4154892018 0 0) 0x7f8bb8003520 con 0x7f8bcc1430c0 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 >> v1:172.21.15.134:6790/0 conn(0x7f8bcc13f710 legacy=0x7f8bcc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 >> v1:172.21.15.134:6789/0 conn(0x7f8bcc0a98f0 legacy=0x7f8bcc13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 1 monclient: found mon.b 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc13afc0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bd35a3640 5 monclient: authenticate success, global_id 4333 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bd35a3640 10 monclient: _renew_subs 2024-03-25T19:31:09.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bd35a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.819+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bcc13b2a0 con 0x7f8bcc1430c0 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.820+0000 7f8bd35a3640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.820+0000 7f8bd35a3640 10 monclient: _renew_subs 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.820+0000 7f8bd35a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.820+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bcc12fa90 con 0x7f8bcc1430c0 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.820+0000 7f8bd35a3640 1 librados: init done 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.821+0000 7f8bd35a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:09.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.821+0000 7f8bd35a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:09.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.821+0000 7f8bd35a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:09.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:09.821+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bcc0b3ef0 con 0x7f8bcc1430c0 2024-03-25T19:31:10.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.091+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc4001fe0 con 0x7f8bcc1430c0 2024-03-25T19:31:10.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.091+0000 7f8bc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:10.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.091+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8bc4003dc0 con 0x7f8bcc1430c0 2024-03-25T19:31:10.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.091+0000 7f8bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:10.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.091+0000 7f8bc1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:10.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.091+0000 7f8bc1ffb640 10 monclient: dump: 2024-03-25T19:31:10.091 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:10.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.091+0000 7f8bc37fe640 4 set_mon_vals no callback set 2024-03-25T19:31:10.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.092+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f8bc4019d90 con 0x7f8bcc1430c0 2024-03-25T19:31:10.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.093+0000 7f8bd133f640 10 monclient: get_auth_request con 0x7f8bb006e160 auth_method 0 2024-03-25T19:31:10.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.093+0000 7f8bc1ffb640 10 client.4333.objecter ms_handle_connect 0x7f8bb006e160 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f8bc4053920 con 0x7f8bcc1430c0 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 10 client.4333.objecter ms_dispatch 0x7f8bcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bc404f430 con 0x7f8bcc1430c0 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:10.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.097+0000 7f8bc1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.348+0000 7f8bd35a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.348+0000 7f8bd35a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.348+0000 7f8bd35a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.348+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bcc0627b0 con 0x7f8bcc1430c0 2024-03-25T19:31:10.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.383+0000 7f8bc1ffb640 1 -- 172.21.15.134:0/2776719884 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f8bc4051360 con 0x7f8bcc1430c0 2024-03-25T19:31:10.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.384+0000 7f8bc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:10.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.384+0000 7f8bc1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:10.384 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:10.384 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.386+0000 7f8bd35a3640 10 librados: watch_flush enter 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.386+0000 7f8bd35a3640 10 librados: watch_flush exit 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.386+0000 7f8bd35a3640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.386+0000 7f8bd35a3640 10 client.4333.objecter successfully canceled tick 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 >> v1:172.21.15.159:6800/1875874887 conn(0x7f8bb006e160 legacy=0x7f8bb0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 10 monclient: shutdown 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 >> v1:172.21.15.159:6789/0 conn(0x7f8bcc1430c0 legacy=0x7f8bcc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:10.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 shutdown_connections 2024-03-25T19:31:10.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 >> 172.21.15.134:0/2776719884 conn(0x7f8bcc0b7ea0 msgr2=0x7f8bcc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:10.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 shutdown_connections 2024-03-25T19:31:10.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 1 -- 172.21.15.134:0/2776719884 wait complete. 2024-03-25T19:31:10.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:10.387+0000 7f8bd35a3640 1 librados: shutdown 2024-03-25T19:31:10.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:17.207 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.325+0000 7fc611339640 10 monclient: get_monmap_and_config 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.325+0000 7fc611339640 10 monclient: build_initial_monmap 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.325+0000 7fc611339640 10 monclient: monmap: 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:17.326215+0000 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:17.326215+0000 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:17.326 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.325+0000 7fc611339640 1 Processor -- start 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.325+0000 7fc611339640 1 -- start start 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.325+0000 7fc611339640 10 monclient: init 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 10 monclient(hunting): picked mon.noname-c con 0x7fc60c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 10 monclient(hunting): picked mon.noname-b con 0x7fc60c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 10 monclient(hunting): picked mon.noname-a con 0x7fc60c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c142520 con 0x7fc60c1430c0 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c0b28a0 con 0x7fc60c0a98f0 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c0b2a30 con 0x7fc60c13f710 2024-03-25T19:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.326+0000 7fc611339640 10 monclient(hunting): authenticate will time out at 6109.485089s 2024-03-25T19:31:17.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.332+0000 7fc60b7fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc60c1430c0 0x7fc60c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:33702/0 (socket says 172.21.15.134:33702) 2024-03-25T19:31:17.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.332+0000 7fc60b7fe640 1 -- 172.21.15.134:0/1891891950 learned_addr learned my addr 172.21.15.134:0/1891891950 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:17.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.339+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998599728 0 0) 0x7fc60c142520 con 0x7fc60c1430c0 2024-03-25T19:31:17.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.340+0000 7fc609ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:17.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.340+0000 7fc609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:17.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.340+0000 7fc609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:17.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.340+0000 7fc609ffb640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:31:17.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.340+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f4003520 con 0x7fc60c1430c0 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558345042 0 0) 0x7fc5f4003520 con 0x7fc60c1430c0 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 >> v1:172.21.15.134:6790/0 conn(0x7fc60c13f710 legacy=0x7fc60c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 >> v1:172.21.15.159:6789/0 conn(0x7fc60c0a98f0 legacy=0x7fc60c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 1 monclient: found mon.noname-a 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c0b44c0 con 0x7fc60c1430c0 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:17.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc604001a10 con 0x7fc60c1430c0 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc611339640 5 monclient: authenticate success, global_id 4368 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc611339640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.344+0000 7fc609ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.345+0000 7fc609ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.345+0000 7fc609ffb640 10 monclient: dump: 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:17.346 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:17.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.345+0000 7fc611339640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc604001b70 con 0x7fc60c1430c0 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc609ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc609ffb640 1 -- 172.21.15.134:0/1891891950 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc604001f80 con 0x7fc60c1430c0 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc611339640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc611339640 4 set_mon_vals no callback set 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc609ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc609ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.349+0000 7fc609ffb640 10 monclient: dump: 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:17.350 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:17.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.350+0000 7fc611339640 10 monclient: shutdown 2024-03-25T19:31:17.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.350+0000 7fc611339640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.350+0000 7fc611339640 1 -- 172.21.15.134:0/1891891950 >> v1:172.21.15.134:6789/0 conn(0x7fc60c1430c0 legacy=0x7fc60c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.350+0000 7fc611339640 1 -- 172.21.15.134:0/1891891950 shutdown_connections 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.350+0000 7fc611339640 1 -- 172.21.15.134:0/1891891950 >> 172.21.15.134:0/1891891950 conn(0x7fc60c0b7ea0 msgr2=0x7fc60c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.350+0000 7fc611339640 1 -- 172.21.15.134:0/1891891950 shutdown_connections 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.350+0000 7fc611339640 1 -- 172.21.15.134:0/1891891950 wait complete. 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.351+0000 7fc611339640 10 monclient: build_initial_monmap 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.351+0000 7fc611339640 10 monclient: monmap: 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:17.352 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.351+0000 7fc611339640 1 librados: starting msgr at 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.351+0000 7fc611339640 1 librados: starting objecter 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.351+0000 7fc611339640 1 Processor -- start 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 1 -- start start 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 1 librados: setting wanted keys 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 1 librados: calling monclient init 2024-03-25T19:31:17.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient: init 2024-03-25T19:31:17.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:17.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:17.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient(hunting): picked mon.noname-c con 0x7fc60c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:17.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient(hunting): picked mon.noname-a con 0x7fc60c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:17.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient(hunting): picked mon.noname-b con 0x7fc60c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:17.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c06fbb0 con 0x7fc60c13f710 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c06fe20 con 0x7fc60c1430c0 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c070100 con 0x7fc60c0a98f0 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.352+0000 7fc611339640 10 monclient(hunting): authenticate will time out at 6109.511440s 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.353+0000 7fc60affd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc60c0a98f0 0x7fc60c070be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:49764/0 (socket says 172.21.15.134:49764) 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.353+0000 7fc60affd640 1 -- 172.21.15.134:0/367485404 learned_addr learned my addr 172.21.15.134:0/367485404 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.353+0000 7fc5f37fe640 10 client.?.objecter ms_handle_connect 0x7fc60c0a98f0 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.353+0000 7fc5f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.354+0000 7fc5f37fe640 10 client.?.objecter ms_handle_connect 0x7fc60c1430c0 2024-03-25T19:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.354+0000 7fc5f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:17.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.369+0000 7fc5f37fe640 10 client.?.objecter ms_handle_connect 0x7fc60c13f710 2024-03-25T19:31:17.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.369+0000 7fc5f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:17.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.439+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188932901 0 0) 0x7fc60c06fe20 con 0x7fc60c1430c0 2024-03-25T19:31:17.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.439+0000 7fc5f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:17.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.439+0000 7fc5f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:17.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.439+0000 7fc5f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:17.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.439+0000 7fc5f37fe640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:31:17.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.439+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f8004eb0 con 0x7fc60c1430c0 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929625558 0 0) 0x7fc60c070100 con 0x7fc60c0a98f0 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc60c06fe20 con 0x7fc60c0a98f0 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3685298286 0 0) 0x7fc5f8004eb0 con 0x7fc60c1430c0 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc60c070100 con 0x7fc60c1430c0 2024-03-25T19:31:17.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc604001e80 con 0x7fc60c1430c0 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): dump: 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:17.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.448+0000 7fc5f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:17.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.470+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239344620 0 0) 0x7fc60c06fbb0 con 0x7fc60c13f710 2024-03-25T19:31:17.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:17.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:17.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:17.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:31:17.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f8004eb0 con 0x7fc60c13f710 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3836102249 0 0) 0x7fc60c070100 con 0x7fc60c1430c0 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 >> v1:172.21.15.134:6790/0 conn(0x7fc60c0a98f0 legacy=0x7fc60c070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 >> v1:172.21.15.134:6789/0 conn(0x7fc60c13f710 legacy=0x7fc60c0712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 1 monclient: found mon.b 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:17.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c0703e0 con 0x7fc60c1430c0 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc5f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 10 monclient: _renew_subs 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c06f240 con 0x7fc60c1430c0 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 10 monclient: _renew_subs 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c06e2f0 con 0x7fc60c1430c0 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.471+0000 7fc611339640 1 librados: init done 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.472+0000 7fc611339640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.472+0000 7fc611339640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:17.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.472+0000 7fc611339640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:17.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.472+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c0627b0 con 0x7fc60c1430c0 2024-03-25T19:31:17.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.748+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc604004e10 con 0x7fc60c1430c0 2024-03-25T19:31:17.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.748+0000 7fc5f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:17.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.748+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc604002100 con 0x7fc60c1430c0 2024-03-25T19:31:17.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.748+0000 7fc5f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.748+0000 7fc5f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.748+0000 7fc5f37fe640 10 monclient: dump: 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:17.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.748+0000 7fc6097fa640 4 set_mon_vals no callback set 2024-03-25T19:31:17.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.749+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fc60401afd0 con 0x7fc60c1430c0 2024-03-25T19:31:17.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.749+0000 7fc60affd640 10 monclient: get_auth_request con 0x7fc5f806dd60 auth_method 0 2024-03-25T19:31:17.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.749+0000 7fc5f37fe640 10 client.4336.objecter ms_handle_connect 0x7fc5f806dd60 2024-03-25T19:31:17.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fc604054c90 con 0x7fc60c1430c0 2024-03-25T19:31:17.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 10 client.4336.objecter ms_dispatch 0x7fc60c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:17.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:17.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:17.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T19:31:17.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6040505d0 con 0x7fc60c1430c0 2024-03-25T19:31:17.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:17.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:17.754+0000 7fc5f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:18.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.009+0000 7fc611339640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:18.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.009+0000 7fc611339640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:18.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.009+0000 7fc611339640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:18.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.009+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc60c06f3d0 con 0x7fc60c1430c0 2024-03-25T19:31:18.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.032+0000 7fc5f37fe640 1 -- 172.21.15.134:0/367485404 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fc6040526d0 con 0x7fc60c1430c0 2024-03-25T19:31:18.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.032+0000 7fc5f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:18.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.032+0000 7fc5f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:18.032 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:18.033 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 10 librados: watch_flush enter 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 10 librados: watch_flush exit 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 10 client.4336.objecter successfully canceled tick 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 >> v1:172.21.15.159:6800/1875874887 conn(0x7fc5f806dd60 legacy=0x7fc5f8070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 10 monclient: shutdown 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 >> v1:172.21.15.159:6789/0 conn(0x7fc60c1430c0 legacy=0x7fc60c1e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 shutdown_connections 2024-03-25T19:31:18.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 >> 172.21.15.134:0/367485404 conn(0x7fc60c0b7ea0 msgr2=0x7fc60c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:18.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 shutdown_connections 2024-03-25T19:31:18.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 1 -- 172.21.15.134:0/367485404 wait complete. 2024-03-25T19:31:18.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:18.034+0000 7fc611339640 1 librados: shutdown 2024-03-25T19:31:18.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:20.009 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:24.681 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.018 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:24.691 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.050 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:24.723 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.074 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:24.747 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.081 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:24.753 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.084 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:24.731 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.092 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:24.739 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.124 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:24.771 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:20.126 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:24.799 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:24.852 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:24.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.964+0000 7f60ffd54640 10 monclient: get_monmap_and_config 2024-03-25T19:31:24.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.964+0000 7f60ffd54640 10 monclient: build_initial_monmap 2024-03-25T19:31:24.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.964+0000 7f60ffd54640 10 monclient: monmap: 2024-03-25T19:31:24.964 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:24.964 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:24.965344+0000 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:24.965344+0000 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:24.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.964+0000 7f60ffd54640 1 Processor -- start 2024-03-25T19:31:24.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.965+0000 7f60ffd54640 1 -- start start 2024-03-25T19:31:24.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.965+0000 7f60ffd54640 10 monclient: init 2024-03-25T19:31:24.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:24.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:24.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 10 monclient(hunting): picked mon.noname-b con 0x7f60f80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:24.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 10 monclient(hunting): picked mon.noname-c con 0x7f60f813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 10 monclient(hunting): picked mon.noname-a con 0x7f60f81430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8142520 con 0x7f60f81430c0 2024-03-25T19:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80b28a0 con 0x7f60f813f710 2024-03-25T19:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80b2a30 con 0x7f60f80a98f0 2024-03-25T19:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.966+0000 7f60ffd54640 10 monclient(hunting): authenticate will time out at 6117.124794s 2024-03-25T19:31:24.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.979+0000 7f60fdaf0640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f60f80a98f0 0x7f60f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:34038/0 (socket says 172.21.15.134:34038) 2024-03-25T19:31:24.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.979+0000 7f60fdaf0640 1 -- 172.21.15.134:0/1602649799 learned_addr learned my addr 172.21.15.134:0/1602649799 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:24.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.994+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958991011 0 0) 0x7f60f80b2a30 con 0x7f60f80a98f0 2024-03-25T19:31:24.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.994+0000 7f60fcaee640 10 monclient(hunting): _negotiate 2024-03-25T19:31:24.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.994+0000 7f60fcaee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:24.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.994+0000 7f60fcaee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:24.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.994+0000 7f60fcaee640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:31:24.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:24.994+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e4003520 con 0x7f60f80a98f0 2024-03-25T19:31:25.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160544717 0 0) 0x7f60f8142520 con 0x7f60f81430c0 2024-03-25T19:31:25.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): _negotiate 2024-03-25T19:31:25.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:25.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:25.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:31:25.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f80b2a30 con 0x7f60f81430c0 2024-03-25T19:31:25.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041641123 0 0) 0x7f60f80b28a0 con 0x7f60f813f710 2024-03-25T19:31:25.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): _negotiate 2024-03-25T19:31:25.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:25.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:25.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:31:25.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.000+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f8142520 con 0x7f60f813f710 2024-03-25T19:31:25.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953264532 0 0) 0x7f60e4003520 con 0x7f60f80a98f0 2024-03-25T19:31:25.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 >> v1:172.21.15.159:6789/0 conn(0x7f60f813f710 legacy=0x7f60f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:25.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 >> v1:172.21.15.134:6789/0 conn(0x7f60f81430c0 legacy=0x7f60f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 10 monclient: _finish_hunting 0 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 1 monclient: found mon.noname-b 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f80b44c0 con 0x7f60f80a98f0 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 10 monclient: _finish_auth 0 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 10 monclient: _check_auth_tickets 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f60f4001690 con 0x7f60f80a98f0 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60ffd54640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60fcaee640 10 monclient: dump: 2024-03-25T19:31:25.017 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:25.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.016+0000 7f60ffd54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:25.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f40017f0 con 0x7f60f80a98f0 2024-03-25T19:31:25.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60fcaee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:25.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60fcaee640 1 -- 172.21.15.134:0/1602649799 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f60f4001c00 con 0x7f60f80a98f0 2024-03-25T19:31:25.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60ffd54640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:25.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60ffd54640 4 set_mon_vals no callback set 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60fcaee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60fcaee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.036+0000 7f60fcaee640 10 monclient: dump: 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:25.037 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:25.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.037+0000 7f60ffd54640 10 monclient: shutdown 2024-03-25T19:31:25.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.037+0000 7f60ffd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.037+0000 7f60ffd54640 1 -- 172.21.15.134:0/1602649799 >> v1:172.21.15.134:6790/0 conn(0x7f60f80a98f0 legacy=0x7f60f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.037+0000 7f60ffd54640 1 -- 172.21.15.134:0/1602649799 shutdown_connections 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.037+0000 7f60ffd54640 1 -- 172.21.15.134:0/1602649799 >> 172.21.15.134:0/1602649799 conn(0x7f60f80b7ea0 msgr2=0x7f60f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.037+0000 7f60ffd54640 1 -- 172.21.15.134:0/1602649799 shutdown_connections 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.037+0000 7f60ffd54640 1 -- 172.21.15.134:0/1602649799 wait complete. 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.038+0000 7f60ffd54640 10 monclient: build_initial_monmap 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.038+0000 7f60ffd54640 10 monclient: monmap: 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:25.040 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.038+0000 7f60ffd54640 1 librados: starting msgr at 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 librados: starting objecter 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 Processor -- start 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 -- start start 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 librados: setting wanted keys 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 librados: calling monclient init 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient: init 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient(hunting): picked mon.noname-a con 0x7f60f80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:25.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient(hunting): picked mon.noname-c con 0x7f60f813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:25.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient(hunting): picked mon.noname-b con 0x7f60f81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:25.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8138b20 con 0x7f60f80a98f0 2024-03-25T19:31:25.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8137a80 con 0x7f60f81430c0 2024-03-25T19:31:25.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8139880 con 0x7f60f813f710 2024-03-25T19:31:25.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:25.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.039+0000 7f60ffd54640 10 monclient(hunting): authenticate will time out at 6117.198623s 2024-03-25T19:31:25.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.047+0000 7f60fe2f1640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f60f81430c0 0x7f60f81e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:48644/0 (socket says 172.21.15.134:48644) 2024-03-25T19:31:25.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.047+0000 7f60fe2f1640 1 -- 172.21.15.134:0/918138810 learned_addr learned my addr 172.21.15.134:0/918138810 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:25.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.048+0000 7f60ee7fc640 10 client.?.objecter ms_handle_connect 0x7f60f81430c0 2024-03-25T19:31:25.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.048+0000 7f60ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:25.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.053+0000 7f60ee7fc640 10 client.?.objecter ms_handle_connect 0x7f60f813f710 2024-03-25T19:31:25.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.053+0000 7f60ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.061+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468439240 0 0) 0x7f60f8137a80 con 0x7f60f81430c0 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.061+0000 7f60ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.061+0000 7f60ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.061+0000 7f60ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.061+0000 7f60ee7fc640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.061+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60dc004eb0 con 0x7f60f81430c0 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.062+0000 7f60ee7fc640 10 client.?.objecter ms_handle_connect 0x7f60f80a98f0 2024-03-25T19:31:25.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.062+0000 7f60ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:25.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.071+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684757097 0 0) 0x7f60f8139880 con 0x7f60f813f710 2024-03-25T19:31:25.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.071+0000 7f60ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:25.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.071+0000 7f60ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:25.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.071+0000 7f60ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:25.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.071+0000 7f60ee7fc640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:31:25.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.071+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f8137a80 con 0x7f60f813f710 2024-03-25T19:31:25.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.076+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085734412 0 0) 0x7f60f8138b20 con 0x7f60f80a98f0 2024-03-25T19:31:25.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.076+0000 7f60ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:25.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.076+0000 7f60ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:25.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.076+0000 7f60ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:25.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.076+0000 7f60ee7fc640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:31:25.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.076+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f8139880 con 0x7f60f80a98f0 2024-03-25T19:31:25.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.080+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 182109239 0 0) 0x7f60f8137a80 con 0x7f60f813f710 2024-03-25T19:31:25.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.080+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60f8138b20 con 0x7f60f813f710 2024-03-25T19:31:25.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.084+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2273991154 0 0) 0x7f60dc004eb0 con 0x7f60f81430c0 2024-03-25T19:31:25.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.084+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60f8137a80 con 0x7f60f81430c0 2024-03-25T19:31:25.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.084+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f60f0001c90 con 0x7f60f81430c0 2024-03-25T19:31:25.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.084+0000 7f60ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:25.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.084+0000 7f60ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.084+0000 7f60ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:25.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.084+0000 7f60ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:25.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.087+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f60e8001930 con 0x7f60f813f710 2024-03-25T19:31:25.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.087+0000 7f60ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:25.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.087+0000 7f60ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.087+0000 7f60ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:25.087 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:25.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2148433648 0 0) 0x7f60f8137a80 con 0x7f60f81430c0 2024-03-25T19:31:25.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 >> v1:172.21.15.134:6790/0 conn(0x7f60f813f710 legacy=0x7f60f8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:25.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 >> v1:172.21.15.134:6789/0 conn(0x7f60f80a98f0 legacy=0x7f60f8137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:25.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:25.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 1 monclient: found mon.b 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f8139a10 con 0x7f60f81430c0 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ffd54640 5 monclient: authenticate success, global_id 4342 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ffd54640 10 monclient: _renew_subs 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ffd54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.096+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f81dab00 con 0x7f60f81430c0 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.097+0000 7f60ffd54640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.097+0000 7f60ffd54640 10 monclient: _renew_subs 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.097+0000 7f60ffd54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:25.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.097+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f813b4f0 con 0x7f60f81430c0 2024-03-25T19:31:25.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.097+0000 7f60ffd54640 1 librados: init done 2024-03-25T19:31:25.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.097+0000 7f60ffd54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:25.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.098+0000 7f60ffd54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:25.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.098+0000 7f60ffd54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:25.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.098+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f80b3e80 con 0x7f60f81430c0 2024-03-25T19:31:25.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.101+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f0003970 con 0x7f60f81430c0 2024-03-25T19:31:25.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.101+0000 7f60ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:25.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.101+0000 7f60effff640 4 set_mon_vals no callback set 2024-03-25T19:31:25.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.369+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f60f0003d80 con 0x7f60f81430c0 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.369+0000 7f60ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.369+0000 7f60ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.369+0000 7f60ee7fc640 10 monclient: dump: 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:25.370 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:25.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.370+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f60f0003f60 con 0x7f60f81430c0 2024-03-25T19:31:25.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.371+0000 7f60fdaf0640 10 monclient: get_auth_request con 0x7f60dc06df50 auth_method 0 2024-03-25T19:31:25.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.371+0000 7f60ee7fc640 10 client.4342.objecter ms_handle_connect 0x7f60dc06df50 2024-03-25T19:31:25.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.373+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f60f0054b00 con 0x7f60f81430c0 2024-03-25T19:31:25.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.373+0000 7f60ee7fc640 10 client.4342.objecter ms_dispatch 0x7f60f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:25.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.373+0000 7f60ee7fc640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:25.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.373+0000 7f60ee7fc640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:25.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.373+0000 7f60ee7fc640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T19:31:25.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.375+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60f0050610 con 0x7f60f81430c0 2024-03-25T19:31:25.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.375+0000 7f60ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:25.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.375+0000 7f60ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:25.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.594+0000 7f60ffd54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:25.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.594+0000 7f60ffd54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:25.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.594+0000 7f60ffd54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:25.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.594+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60f80627b0 con 0x7f60f81430c0 2024-03-25T19:31:25.613 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.613+0000 7f60ee7fc640 1 -- 172.21.15.134:0/918138810 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f60f0052540 con 0x7f60f81430c0 2024-03-25T19:31:25.613 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.613+0000 7f60ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:25.613 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.613+0000 7f60ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:25.614 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:25.614 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 10 librados: watch_flush enter 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 10 librados: watch_flush exit 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 10 client.4342.objecter successfully canceled tick 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 >> v1:172.21.15.159:6800/1875874887 conn(0x7f60dc06df50 legacy=0x7f60dc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 10 monclient: shutdown 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 >> v1:172.21.15.159:6789/0 conn(0x7f60f81430c0 legacy=0x7f60f81e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 shutdown_connections 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 >> 172.21.15.134:0/918138810 conn(0x7f60f80b7ea0 msgr2=0x7f60f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 shutdown_connections 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 1 -- 172.21.15.134:0/918138810 wait complete. 2024-03-25T19:31:25.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:25.616+0000 7f60ffd54640 1 librados: shutdown 2024-03-25T19:31:25.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:32.434 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.534+0000 7f85ac12b640 10 monclient: get_monmap_and_config 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.534+0000 7f85ac12b640 10 monclient: build_initial_monmap 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.534+0000 7f85ac12b640 10 monclient: monmap: 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:32.535331+0000 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:32.535331+0000 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:32.535 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:32.535 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:32.535 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.534+0000 7f85ac12b640 1 Processor -- start 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 1 -- start start 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient: init 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient(hunting): picked mon.noname-b con 0x7f85a406c380 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient(hunting): picked mon.noname-a con 0x7f85a406d670 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient(hunting): picked mon.noname-c con 0x7f85a40af8f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:32.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a406c9a0 con 0x7f85a406d670 2024-03-25T19:31:32.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a40ada90 con 0x7f85a40af8f0 2024-03-25T19:31:32.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a40b0d10 con 0x7f85a406c380 2024-03-25T19:31:32.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:32.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.535+0000 7f85ac12b640 10 monclient(hunting): authenticate will time out at 6124.694334s 2024-03-25T19:31:32.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.548+0000 7f85a96c6640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f85a406d670 0x7f85a40ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:44266/0 (socket says 172.21.15.134:44266) 2024-03-25T19:31:32.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.548+0000 7f85a96c6640 1 -- 172.21.15.134:0/766618351 learned_addr learned my addr 172.21.15.134:0/766618351 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:32.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.568+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868254964 0 0) 0x7f85a406c9a0 con 0x7f85a406d670 2024-03-25T19:31:32.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.568+0000 7f85a8ec5640 10 monclient(hunting): _negotiate 2024-03-25T19:31:32.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.568+0000 7f85a8ec5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:32.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.568+0000 7f85a8ec5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:32.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.568+0000 7f85a8ec5640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:31:32.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.568+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8590003520 con 0x7f85a406d670 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.583+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757099 0 0) 0x7f8590003520 con 0x7f85a406d670 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.583+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 >> v1:172.21.15.134:6790/0 conn(0x7f85a406c380 legacy=0x7f85a40b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 >> v1:172.21.15.159:6789/0 conn(0x7f85a40af8f0 legacy=0x7f85a40aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 10 monclient: _finish_hunting 0 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 1 monclient: found mon.noname-a 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a40add00 con 0x7f85a406d670 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 10 monclient: _finish_auth 0 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 10 monclient: _check_auth_tickets 2024-03-25T19:31:32.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8594001690 con 0x7f85a406d670 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85a8ec5640 10 monclient: dump: 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85ac12b640 5 monclient: authenticate success, global_id 4380 2024-03-25T19:31:32.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.584+0000 7f85ac12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:32.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85940017f0 con 0x7f85a406d670 2024-03-25T19:31:32.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85a8ec5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85a8ec5640 1 -- 172.21.15.134:0/766618351 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8594001c40 con 0x7f85a406d670 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85ac12b640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85ac12b640 4 set_mon_vals no callback set 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85a8ec5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85a8ec5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.619+0000 7f85a8ec5640 10 monclient: dump: 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:32.621 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:32.621 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:32.621 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:32.621 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:32.621 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:32.621 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:32.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.620+0000 7f85ac12b640 10 monclient: shutdown 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.620+0000 7f85ac12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.620+0000 7f85ac12b640 1 -- 172.21.15.134:0/766618351 >> v1:172.21.15.134:6789/0 conn(0x7f85a406d670 legacy=0x7f85a40ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.620+0000 7f85ac12b640 1 -- 172.21.15.134:0/766618351 shutdown_connections 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.620+0000 7f85ac12b640 1 -- 172.21.15.134:0/766618351 >> 172.21.15.134:0/766618351 conn(0x7f85a40b5c00 msgr2=0x7f85a40b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.620+0000 7f85ac12b640 1 -- 172.21.15.134:0/766618351 shutdown_connections 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.620+0000 7f85ac12b640 1 -- 172.21.15.134:0/766618351 wait complete. 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 10 monclient: build_initial_monmap 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 10 monclient: monmap: 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:32.624 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 1 librados: starting msgr at 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 1 librados: starting objecter 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 1 Processor -- start 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 1 -- start start 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 1 librados: setting wanted keys 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 1 librados: calling monclient init 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.621+0000 7f85ac12b640 10 monclient: init 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 10 monclient(hunting): picked mon.noname-c con 0x7f85a406c380 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:32.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 10 monclient(hunting): picked mon.noname-a con 0x7f85a406d670 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 10 monclient(hunting): picked mon.noname-b con 0x7f85a40af8f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a4149300 con 0x7f85a406d670 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a413fa30 con 0x7f85a40af8f0 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a4149550 con 0x7f85a406c380 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85ac12b640 10 monclient(hunting): authenticate will time out at 6124.781147s 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85a9ec7640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f85a406c380 0x7f85a413ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50982/0 (socket says 172.21.15.134:50982) 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.622+0000 7f85a9ec7640 1 -- 172.21.15.134:0/1233403240 learned_addr learned my addr 172.21.15.134:0/1233403240 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.623+0000 7f859a7fc640 10 client.?.objecter ms_handle_connect 0x7f85a406c380 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.623+0000 7f859a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.623+0000 7f859a7fc640 10 client.?.objecter ms_handle_connect 0x7f85a40af8f0 2024-03-25T19:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.623+0000 7f859a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:32.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.669+0000 7f859a7fc640 10 client.?.objecter ms_handle_connect 0x7f85a406d670 2024-03-25T19:31:32.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.669+0000 7f859a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:32.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.682+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258985361 0 0) 0x7f85a4149300 con 0x7f85a406d670 2024-03-25T19:31:32.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.682+0000 7f859a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:32.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.682+0000 7f859a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:32.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.682+0000 7f859a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:32.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.682+0000 7f859a7fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:31:32.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.682+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8588004eb0 con 0x7f85a406d670 2024-03-25T19:31:32.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.693+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529380925 0 0) 0x7f85a413fa30 con 0x7f85a40af8f0 2024-03-25T19:31:32.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.693+0000 7f859a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:32.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.693+0000 7f859a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:32.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.693+0000 7f859a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:32.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.693+0000 7f859a7fc640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:31:32.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.693+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a4149300 con 0x7f85a40af8f0 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.694+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717845653 0 0) 0x7f8588004eb0 con 0x7f85a406d670 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.694+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85a413fa30 con 0x7f85a406d670 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.694+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8594001b00 con 0x7f85a406d670 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.694+0000 7f859a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.694+0000 7f859a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.694+0000 7f859a7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:32.694 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:32.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.694+0000 7f859a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2110654657 0 0) 0x7f85a413fa30 con 0x7f85a406d670 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 >> v1:172.21.15.134:6790/0 conn(0x7f85a406c380 legacy=0x7f85a413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 >> v1:172.21.15.159:6789/0 conn(0x7f85a40af8f0 legacy=0x7f85a413f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 1 monclient: found mon.a 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a4140f70 con 0x7f85a406d670 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:32.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.704+0000 7f859a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 5 monclient: authenticate success, global_id 4383 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 10 monclient: _renew_subs 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a4144610 con 0x7f85a406d670 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 10 monclient: _renew_subs 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a4140ba0 con 0x7f85a406d670 2024-03-25T19:31:32.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.705+0000 7f85ac12b640 1 librados: init done 2024-03-25T19:31:32.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.706+0000 7f85ac12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:32.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.706+0000 7f85ac12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:32.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.706+0000 7f85ac12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:32.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.706+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a40b6560 con 0x7f85a406d670 2024-03-25T19:31:32.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.999+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8594001c80 con 0x7f85a406d670 2024-03-25T19:31:32.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.999+0000 7f859a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:32.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.999+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8594004d90 con 0x7f85a406d670 2024-03-25T19:31:32.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.999+0000 7f859a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:32.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.999+0000 7f859a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:32.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.999+0000 7f859a7fc640 10 monclient: dump: 2024-03-25T19:31:32.999 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:33.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:32.999+0000 7f859bfff640 4 set_mon_vals no callback set 2024-03-25T19:31:33.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.000+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f859401ad60 con 0x7f85a406d670 2024-03-25T19:31:33.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.000+0000 7f85a9ec7640 10 monclient: get_auth_request con 0x7f858806dda0 auth_method 0 2024-03-25T19:31:33.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.001+0000 7f859a7fc640 10 client.4383.objecter ms_handle_connect 0x7f858806dda0 2024-03-25T19:31:33.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f8594054850 con 0x7f85a406d670 2024-03-25T19:31:33.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 10 client.4383.objecter ms_dispatch 0x7f85a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:33.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:33.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:33.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-25T19:31:33.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8594050360 con 0x7f85a406d670 2024-03-25T19:31:33.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:33.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.006+0000 7f859a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:33.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.249+0000 7f85ac12b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:33.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.249+0000 7f85ac12b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:33.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.249+0000 7f85ac12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:33.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.249+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85a40af480 con 0x7f85a406d670 2024-03-25T19:31:33.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.267+0000 7f859a7fc640 1 -- 172.21.15.134:0/1233403240 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f8594052290 con 0x7f85a406d670 2024-03-25T19:31:33.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.267+0000 7f859a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:33.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.267+0000 7f859a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:33.267 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:33.267 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:33.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 10 librados: watch_flush enter 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 10 librados: watch_flush exit 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 10 client.4383.objecter successfully canceled tick 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 >> v1:172.21.15.159:6800/1875874887 conn(0x7f858806dda0 legacy=0x7f8588070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 10 monclient: shutdown 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 >> v1:172.21.15.134:6789/0 conn(0x7f85a406d670 legacy=0x7f85a413cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 shutdown_connections 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 >> 172.21.15.134:0/1233403240 conn(0x7f85a40b5c00 msgr2=0x7f85a414e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 shutdown_connections 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.270+0000 7f85ac12b640 1 -- 172.21.15.134:0/1233403240 wait complete. 2024-03-25T19:31:33.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:33.271+0000 7f85ac12b640 1 librados: shutdown 2024-03-25T19:31:33.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:39.994 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:44.666 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.048 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:44.720 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.078 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:44.751 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.087 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:40.103 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:44.776 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.109 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:44.781 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.112 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:44.759 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.116 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:44.763 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.138 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:44.810 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.148 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:44.795 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:40.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.192+0000 7f83e8513640 10 monclient: get_monmap_and_config 2024-03-25T19:31:40.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.192+0000 7f83e8513640 10 monclient: build_initial_monmap 2024-03-25T19:31:40.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.192+0000 7f83e8513640 10 monclient: monmap: 2024-03-25T19:31:40.192 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:40.193852+0000 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:40.193852+0000 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:40.193 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.193+0000 7f83e8513640 1 Processor -- start 2024-03-25T19:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.193+0000 7f83e8513640 1 -- start start 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.193+0000 7f83e8513640 10 monclient: init 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 10 monclient(hunting): picked mon.noname-c con 0x7f83e00b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 10 monclient(hunting): picked mon.noname-b con 0x7f83e00b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 10 monclient(hunting): picked mon.noname-a con 0x7f83e014be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e00af940 con 0x7f83e014be90 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e00b0a70 con 0x7f83e00b4d60 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e00b0d50 con 0x7f83e00b1730 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:40.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.194+0000 7f83e8513640 10 monclient(hunting): authenticate will time out at 6132.352898s 2024-03-25T19:31:40.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.201+0000 7f83e62af640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f83e00b4d60 0x7f83e013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:55330/0 (socket says 172.21.15.134:55330) 2024-03-25T19:31:40.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.201+0000 7f83e62af640 1 -- 172.21.15.134:0/693183893 learned_addr learned my addr 172.21.15.134:0/693183893 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.216+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102874661 0 0) 0x7f83e00b0a70 con 0x7f83e00b4d60 2024-03-25T19:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.216+0000 7f83e52ad640 10 monclient(hunting): _negotiate 2024-03-25T19:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.216+0000 7f83e52ad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.216+0000 7f83e52ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.216+0000 7f83e52ad640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.216+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83cc003520 con 0x7f83e00b4d60 2024-03-25T19:31:40.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.243+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051068150 0 0) 0x7f83e00af940 con 0x7f83e014be90 2024-03-25T19:31:40.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient(hunting): _negotiate 2024-03-25T19:31:40.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:40.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:40.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:31:40.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e00b0a70 con 0x7f83e014be90 2024-03-25T19:31:40.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312518559 0 0) 0x7f83cc003520 con 0x7f83e00b4d60 2024-03-25T19:31:40.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 >> v1:172.21.15.134:6790/0 conn(0x7f83e00b1730 legacy=0x7f83e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:40.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 >> v1:172.21.15.134:6789/0 conn(0x7f83e014be90 legacy=0x7f83e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient: _finish_hunting 0 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 1 monclient: found mon.noname-c 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e00b1030 con 0x7f83e00b4d60 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient: _finish_auth 0 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient: _check_auth_tickets 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f83dc001690 con 0x7f83e00b4d60 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e8513640 5 monclient: authenticate success, global_id 4348 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e8513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:40.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e52ad640 10 monclient: dump: 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:40.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.244+0000 7f83e8513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:40.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83dc0017f0 con 0x7f83e00b4d60 2024-03-25T19:31:40.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e52ad640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:40.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e52ad640 1 -- 172.21.15.134:0/693183893 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f83dc001c40 con 0x7f83e00b4d60 2024-03-25T19:31:40.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e52ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:40.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e52ad640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e52ad640 10 monclient: dump: 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e8513640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:40.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.261+0000 7f83e8513640 4 set_mon_vals no callback set 2024-03-25T19:31:40.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.262+0000 7f83e8513640 10 monclient: shutdown 2024-03-25T19:31:40.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.262+0000 7f83e8513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.262+0000 7f83e8513640 1 -- 172.21.15.134:0/693183893 >> v1:172.21.15.159:6789/0 conn(0x7f83e00b4d60 legacy=0x7f83e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.262+0000 7f83e8513640 1 -- 172.21.15.134:0/693183893 shutdown_connections 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.262+0000 7f83e8513640 1 -- 172.21.15.134:0/693183893 >> 172.21.15.134:0/693183893 conn(0x7f83e00617a0 msgr2=0x7f83e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.262+0000 7f83e8513640 1 -- 172.21.15.134:0/693183893 shutdown_connections 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.262+0000 7f83e8513640 1 -- 172.21.15.134:0/693183893 wait complete. 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 10 monclient: build_initial_monmap 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 10 monclient: monmap: 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 1 librados: starting msgr at 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 1 librados: starting objecter 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 1 Processor -- start 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 1 -- start start 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 1 librados: setting wanted keys 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 1 librados: calling monclient init 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.263+0000 7f83e8513640 10 monclient: init 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 10 monclient(hunting): picked mon.noname-b con 0x7f83e00b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 10 monclient(hunting): picked mon.noname-a con 0x7f83e00b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 10 monclient(hunting): picked mon.noname-c con 0x7f83e014be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e0141ce0 con 0x7f83e00b4d60 2024-03-25T19:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e0141fc0 con 0x7f83e00b1730 2024-03-25T19:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e013bc70 con 0x7f83e014be90 2024-03-25T19:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.264+0000 7f83e8513640 10 monclient(hunting): authenticate will time out at 6132.423340s 2024-03-25T19:31:40.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.280+0000 7f83e5aae640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f83e00b4d60 0x7f83e01423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49172/0 (socket says 172.21.15.134:49172) 2024-03-25T19:31:40.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.280+0000 7f83e5aae640 1 -- 172.21.15.134:0/2121291339 learned_addr learned my addr 172.21.15.134:0/2121291339 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:40.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.281+0000 7f83d6ffd640 10 client.?.objecter ms_handle_connect 0x7f83e00b4d60 2024-03-25T19:31:40.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.281+0000 7f83d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:40.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.284+0000 7f83d6ffd640 10 client.?.objecter ms_handle_connect 0x7f83e00b1730 2024-03-25T19:31:40.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.284+0000 7f83d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:40.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.291+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231785138 0 0) 0x7f83e0141ce0 con 0x7f83e00b4d60 2024-03-25T19:31:40.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.291+0000 7f83d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:40.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.291+0000 7f83d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:40.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.291+0000 7f83d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:40.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.291+0000 7f83d6ffd640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:31:40.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.291+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c4004eb0 con 0x7f83e00b4d60 2024-03-25T19:31:40.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.299+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854762091 0 0) 0x7f83e0141fc0 con 0x7f83e00b1730 2024-03-25T19:31:40.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.299+0000 7f83d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:40.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.299+0000 7f83d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:40.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.299+0000 7f83d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:40.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.299+0000 7f83d6ffd640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:31:40.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.299+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e0141ce0 con 0x7f83e00b1730 2024-03-25T19:31:40.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.302+0000 7f83d6ffd640 10 client.?.objecter ms_handle_connect 0x7f83e014be90 2024-03-25T19:31:40.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.302+0000 7f83d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.316+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1754940900 0 0) 0x7f83c4004eb0 con 0x7f83e00b4d60 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.316+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83e0141fc0 con 0x7f83e00b4d60 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.316+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f83d0001990 con 0x7f83e00b4d60 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.316+0000 7f83d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.316+0000 7f83d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.316+0000 7f83d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.316 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.317 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:40.317 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:40.317 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:40.317 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:40.317 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:40.317 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:40.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.316+0000 7f83d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:40.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.322+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3195104388 0 0) 0x7f83e0141ce0 con 0x7f83e00b1730 2024-03-25T19:31:40.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.322+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83c4004eb0 con 0x7f83e00b1730 2024-03-25T19:31:40.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.322+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f83dc001b00 con 0x7f83e00b1730 2024-03-25T19:31:40.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.322+0000 7f83d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.322+0000 7f83d6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.322+0000 7f83d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:40.323 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:40.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.326+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066718729 0 0) 0x7f83e013bc70 con 0x7f83e014be90 2024-03-25T19:31:40.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.326+0000 7f83d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:40.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.326+0000 7f83d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:40.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.326+0000 7f83d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:40.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.326+0000 7f83d6ffd640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:31:40.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.326+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e0141ce0 con 0x7f83e014be90 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 784942457 0 0) 0x7f83c4004eb0 con 0x7f83e00b1730 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 >> v1:172.21.15.134:6790/0 conn(0x7f83e014be90 legacy=0x7f83e0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 >> v1:172.21.15.134:6789/0 conn(0x7f83e00b4d60 legacy=0x7f83e01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 1 monclient: found mon.b 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0140760 con 0x7f83e00b1730 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:40.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83e8513640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83e8513640 10 monclient: _renew_subs 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.331+0000 7f83e8513640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.332+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e0140c80 con 0x7f83e00b1730 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.332+0000 7f83e8513640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.332+0000 7f83e8513640 10 monclient: _renew_subs 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.332+0000 7f83e8513640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.332+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e0140380 con 0x7f83e00b1730 2024-03-25T19:31:40.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.332+0000 7f83e8513640 1 librados: init done 2024-03-25T19:31:40.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.333+0000 7f83e8513640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:40.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.333+0000 7f83e8513640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:40.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.333+0000 7f83e8513640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:40.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.333+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e00afbd0 con 0x7f83e00b1730 2024-03-25T19:31:40.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.636+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83dc001240 con 0x7f83e00b1730 2024-03-25T19:31:40.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.636+0000 7f83d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.636+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f83dc004bf0 con 0x7f83e00b1730 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.636+0000 7f83d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.636+0000 7f83d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.636+0000 7f83d6ffd640 10 monclient: dump: 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:40.637 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:40.638 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:40.638 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:40.638 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:40.638 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:40.638 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:40.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.636+0000 7f83e4aac640 4 set_mon_vals no callback set 2024-03-25T19:31:40.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.637+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f83dc01abc0 con 0x7f83e00b1730 2024-03-25T19:31:40.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.638+0000 7f83e5aae640 10 monclient: get_auth_request con 0x7f83c406df50 auth_method 0 2024-03-25T19:31:40.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.638+0000 7f83d6ffd640 10 client.4351.objecter ms_handle_connect 0x7f83c406df50 2024-03-25T19:31:40.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.644+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f83dc054800 con 0x7f83e00b1730 2024-03-25T19:31:40.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.644+0000 7f83d6ffd640 10 client.4351.objecter ms_dispatch 0x7f83e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:40.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.644+0000 7f83d6ffd640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:40.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.644+0000 7f83d6ffd640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:40.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.644+0000 7f83d6ffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:31:40.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.645+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83dc050310 con 0x7f83e00b1730 2024-03-25T19:31:40.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.645+0000 7f83d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:40.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.645+0000 7f83d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:40.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.893+0000 7f83e8513640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:40.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.893+0000 7f83e8513640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:40.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.893+0000 7f83e8513640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:31:40.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.893+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83e01408f0 con 0x7f83e00b1730 2024-03-25T19:31:40.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.916+0000 7f83d6ffd640 1 -- 172.21.15.134:0/2121291339 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f83dc052240 con 0x7f83e00b1730 2024-03-25T19:31:40.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.916+0000 7f83d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:40.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.916+0000 7f83d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:40.917 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:40.917 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:40.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 10 librados: watch_flush enter 2024-03-25T19:31:40.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 10 librados: watch_flush exit 2024-03-25T19:31:40.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 >> v1:172.21.15.159:6800/1875874887 conn(0x7f83c406df50 legacy=0x7f83c4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 10 monclient: shutdown 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 >> v1:172.21.15.159:6789/0 conn(0x7f83e00b1730 legacy=0x7f83e01db200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 shutdown_connections 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 >> 172.21.15.134:0/2121291339 conn(0x7f83e00617a0 msgr2=0x7f83e013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 shutdown_connections 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 1 -- 172.21.15.134:0/2121291339 wait complete. 2024-03-25T19:31:40.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:40.919+0000 7f83e8513640 1 librados: shutdown 2024-03-25T19:31:40.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:47.738 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:47.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.852+0000 7f7aa62a8640 10 monclient: get_monmap_and_config 2024-03-25T19:31:47.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.852+0000 7f7aa62a8640 10 monclient: build_initial_monmap 2024-03-25T19:31:47.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.852+0000 7f7aa62a8640 10 monclient: monmap: 2024-03-25T19:31:47.852 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:47.852 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:47.853682+0000 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:47.853682+0000 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:47.853 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:47.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.853+0000 7f7aa62a8640 1 Processor -- start 2024-03-25T19:31:47.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.853+0000 7f7aa62a8640 1 -- start start 2024-03-25T19:31:47.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.853+0000 7f7aa62a8640 10 monclient: init 2024-03-25T19:31:47.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:47.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa013f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa0142520 con 0x7f7aa01430c0 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa00b28a0 con 0x7f7aa013f710 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa00b2a30 con 0x7f7aa00a98f0 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:47.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.854+0000 7f7aa62a8640 10 monclient(hunting): authenticate will time out at 6140.013244s 2024-03-25T19:31:47.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.862+0000 7f7aa4845640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7aa01430c0 0x7f7aa014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:45162/0 (socket says 172.21.15.134:45162) 2024-03-25T19:31:47.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.862+0000 7f7aa4845640 1 -- 172.21.15.134:0/2344200823 learned_addr learned my addr 172.21.15.134:0/2344200823 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:47.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.883+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394648573 0 0) 0x7f7aa00b2a30 con 0x7f7aa00a98f0 2024-03-25T19:31:47.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.883+0000 7f7a9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:47.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.883+0000 7f7a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:47.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.883+0000 7f7a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:47.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.883+0000 7f7a9effd640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:31:47.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.883+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a88003520 con 0x7f7aa00a98f0 2024-03-25T19:31:47.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.886+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020947685 0 0) 0x7f7aa00b28a0 con 0x7f7aa013f710 2024-03-25T19:31:47.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.886+0000 7f7a9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:47.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.886+0000 7f7a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:47.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.886+0000 7f7a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:47.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.886+0000 7f7a9effd640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.886+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa00b2a30 con 0x7f7aa013f710 2024-03-25T19:31:47.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185963646 0 0) 0x7f7aa0142520 con 0x7f7aa01430c0 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa00b28a0 con 0x7f7aa01430c0 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823795217 0 0) 0x7f7aa00b2a30 con 0x7f7aa013f710 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 >> v1:172.21.15.134:6790/0 conn(0x7f7aa00a98f0 legacy=0x7f7aa00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 >> v1:172.21.15.134:6789/0 conn(0x7f7aa01430c0 legacy=0x7f7aa014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 monclient: found mon.noname-c 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:31:47.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa00b44c0 con 0x7f7aa013f710 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient: _finish_auth 0 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7a94001670 con 0x7f7aa013f710 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7aa62a8640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7aa62a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7a9effd640 10 monclient: dump: 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:47.896 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:47.897 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:47.897 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:47.897 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:47.897 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:47.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.893+0000 7f7aa62a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:47.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a940017d0 con 0x7f7aa013f710 2024-03-25T19:31:47.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7a9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7a9effd640 1 -- 172.21.15.134:0/2344200823 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7a94001be0 con 0x7f7aa013f710 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7a9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7a9effd640 10 monclient: dump: 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:47.922 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:47.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7aa62a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:47.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.921+0000 7f7aa62a8640 4 set_mon_vals no callback set 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.922+0000 7f7aa62a8640 10 monclient: shutdown 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.922+0000 7f7aa62a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.922+0000 7f7aa62a8640 1 -- 172.21.15.134:0/2344200823 >> v1:172.21.15.159:6789/0 conn(0x7f7aa013f710 legacy=0x7f7aa01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.922+0000 7f7aa62a8640 1 -- 172.21.15.134:0/2344200823 shutdown_connections 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.922+0000 7f7aa62a8640 1 -- 172.21.15.134:0/2344200823 >> 172.21.15.134:0/2344200823 conn(0x7f7aa00b7ea0 msgr2=0x7f7aa00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.922+0000 7f7aa62a8640 1 -- 172.21.15.134:0/2344200823 shutdown_connections 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.922+0000 7f7aa62a8640 1 -- 172.21.15.134:0/2344200823 wait complete. 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.923+0000 7f7aa62a8640 10 monclient: build_initial_monmap 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.923+0000 7f7aa62a8640 10 monclient: monmap: 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:47.924 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.924+0000 7f7aa62a8640 1 librados: starting msgr at 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.924+0000 7f7aa62a8640 1 librados: starting objecter 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.924+0000 7f7aa62a8640 1 Processor -- start 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.924+0000 7f7aa62a8640 1 -- start start 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.924+0000 7f7aa62a8640 1 librados: setting wanted keys 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.924+0000 7f7aa62a8640 1 librados: calling monclient init 2024-03-25T19:31:47.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.924+0000 7f7aa62a8640 10 monclient: init 2024-03-25T19:31:47.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7aa00a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa013f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa01430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa00b0e90 con 0x7f7aa00a98f0 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa01d82b0 con 0x7f7aa01430c0 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa01db5b0 con 0x7f7aa013f710 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:47.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.925+0000 7f7aa62a8640 10 monclient(hunting): authenticate will time out at 6140.083791s 2024-03-25T19:31:47.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.927+0000 7f7a9ffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7aa00a98f0 0x7f7aa01d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:45210/0 (socket says 172.21.15.134:45210) 2024-03-25T19:31:47.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.927+0000 7f7a9ffff640 1 -- 172.21.15.134:0/1281454680 learned_addr learned my addr 172.21.15.134:0/1281454680 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:47.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.933+0000 7f7a9cff9640 10 client.?.objecter ms_handle_connect 0x7f7aa00a98f0 2024-03-25T19:31:47.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.933+0000 7f7a9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:47.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.945+0000 7f7a9cff9640 10 client.?.objecter ms_handle_connect 0x7f7aa01430c0 2024-03-25T19:31:47.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.945+0000 7f7a9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:47.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.945+0000 7f7a9cff9640 10 client.?.objecter ms_handle_connect 0x7f7aa013f710 2024-03-25T19:31:47.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.945+0000 7f7a9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:47.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.946+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832447164 0 0) 0x7f7aa00b0e90 con 0x7f7aa00a98f0 2024-03-25T19:31:47.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.946+0000 7f7a9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:47.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.946+0000 7f7a9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:47.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.946+0000 7f7a9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:47.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.946+0000 7f7a9cff9640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:31:47.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.946+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8c004eb0 con 0x7f7aa00a98f0 2024-03-25T19:31:47.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.954+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048618332 0 0) 0x7f7aa01db5b0 con 0x7f7aa013f710 2024-03-25T19:31:47.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.954+0000 7f7a9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:47.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.954+0000 7f7a9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:47.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.954+0000 7f7a9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:47.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.954+0000 7f7a9cff9640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:31:47.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.954+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa00b0e90 con 0x7f7aa013f710 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.956+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14335322 0 0) 0x7f7a8c004eb0 con 0x7f7aa00a98f0 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.956+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7aa01db5b0 con 0x7f7aa00a98f0 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.956+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7a90001990 con 0x7f7aa00a98f0 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.956+0000 7f7a9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.956+0000 7f7a9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.956+0000 7f7a9cff9640 10 monclient(hunting): dump: 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:47.956 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.956+0000 7f7a9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.957+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186360401 0 0) 0x7f7aa01d82b0 con 0x7f7aa01430c0 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.957+0000 7f7a9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.957+0000 7f7a9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.957+0000 7f7a9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.957+0000 7f7a9cff9640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:31:47.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.957+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8c004eb0 con 0x7f7aa01430c0 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.967+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478598707 0 0) 0x7f7aa00b0e90 con 0x7f7aa013f710 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.967+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7aa01d82b0 con 0x7f7aa013f710 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.967+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7a94001ae0 con 0x7f7aa013f710 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.967+0000 7f7a9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.967+0000 7f7a9cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.967+0000 7f7a9cff9640 10 monclient(hunting): dump: 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:47.967 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:47.968 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:47.968 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:47.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 255911777 0 0) 0x7f7aa01db5b0 con 0x7f7aa00a98f0 2024-03-25T19:31:47.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 >> v1:172.21.15.134:6790/0 conn(0x7f7aa013f710 legacy=0x7f7aa00af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:47.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 >> v1:172.21.15.159:6789/0 conn(0x7f7aa01430c0 legacy=0x7f7aa01d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:47.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:47.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 1 monclient: found mon.a 2024-03-25T19:31:47.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa01db740 con 0x7f7aa00a98f0 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7a9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 5 monclient: authenticate success, global_id 4395 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 10 monclient: _renew_subs 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa014df60 con 0x7f7aa00a98f0 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 10 monclient: _renew_subs 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.980+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa00b1420 con 0x7f7aa00a98f0 2024-03-25T19:31:47.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.981+0000 7f7aa62a8640 1 librados: init done 2024-03-25T19:31:47.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.981+0000 7f7aa62a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:47.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.981+0000 7f7aa62a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:47.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.981+0000 7f7aa62a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:47.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:47.981+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa014e300 con 0x7f7aa00a98f0 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.297+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a900035f0 con 0x7f7aa00a98f0 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.297+0000 7f7a9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.297+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7a900039c0 con 0x7f7aa00a98f0 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.297+0000 7f7a9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.297+0000 7f7a9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.297+0000 7f7a9cff9640 10 monclient: dump: 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:48.297 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.297+0000 7f7a9e7fc640 4 set_mon_vals no callback set 2024-03-25T19:31:48.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.298+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f7a90003be0 con 0x7f7aa00a98f0 2024-03-25T19:31:48.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.298+0000 7f7a9f7fe640 10 monclient: get_auth_request con 0x7f7a8c06dfb0 auth_method 0 2024-03-25T19:31:48.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.299+0000 7f7a9cff9640 10 client.4395.objecter ms_handle_connect 0x7f7a8c06dfb0 2024-03-25T19:31:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.303+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f7a900547a0 con 0x7f7aa00a98f0 2024-03-25T19:31:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.303+0000 7f7a9cff9640 10 client.4395.objecter ms_dispatch 0x7f7aa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.303+0000 7f7a9cff9640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.303+0000 7f7a9cff9640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:48.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.303+0000 7f7a9cff9640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-25T19:31:48.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.303+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a900502b0 con 0x7f7aa00a98f0 2024-03-25T19:31:48.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.304+0000 7f7a9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:48.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.304+0000 7f7a9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:48.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.528+0000 7f7aa62a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:48.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.528+0000 7f7aa62a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:48.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.528+0000 7f7aa62a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:31:48.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.528+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7aa01db8d0 con 0x7f7aa00a98f0 2024-03-25T19:31:48.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.562+0000 7f7a9cff9640 1 -- 172.21.15.134:0/1281454680 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f7a900521e0 con 0x7f7aa00a98f0 2024-03-25T19:31:48.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.563+0000 7f7a9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:48.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.563+0000 7f7a9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:48.563 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:48.563 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:48.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 10 librados: watch_flush enter 2024-03-25T19:31:48.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 10 librados: watch_flush exit 2024-03-25T19:31:48.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 10 client.4395.objecter successfully canceled tick 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 >> v1:172.21.15.159:6800/1875874887 conn(0x7f7a8c06dfb0 legacy=0x7f7a8c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 10 monclient: shutdown 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 >> v1:172.21.15.134:6789/0 conn(0x7f7aa00a98f0 legacy=0x7f7aa01d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 shutdown_connections 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 >> 172.21.15.134:0/1281454680 conn(0x7f7aa00b7ea0 msgr2=0x7f7aa013c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 shutdown_connections 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 1 -- 172.21.15.134:0/1281454680 wait complete. 2024-03-25T19:31:48.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:48.565+0000 7f7aa62a8640 1 librados: shutdown 2024-03-25T19:31:48.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:55.383 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:55.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.485+0000 7fe0a6206640 10 monclient: get_monmap_and_config 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.485+0000 7fe0a6206640 10 monclient: build_initial_monmap 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.485+0000 7fe0a6206640 10 monclient: monmap: 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:31:55.486906+0000 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:31:55.486906+0000 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:31:55.486 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.486+0000 7fe0a6206640 1 Processor -- start 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.486+0000 7fe0a6206640 1 -- start start 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.486+0000 7fe0a6206640 10 monclient: init 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a00b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a00b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a014be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00af940 con 0x7fe0a014be90 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00b0a70 con 0x7fe0a00b4d60 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00b0d50 con 0x7fe0a00b1730 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:55.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.487+0000 7fe0a6206640 10 monclient(hunting): authenticate will time out at 6147.646319s 2024-03-25T19:31:55.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.496+0000 7fe09f7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe0a00b4d60 0x7fe0a013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:59314/0 (socket says 172.21.15.134:59314) 2024-03-25T19:31:55.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.496+0000 7fe09f7fe640 1 -- 172.21.15.134:0/3901288799 learned_addr learned my addr 172.21.15.134:0/3901288799 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:55.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.503+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256855368 0 0) 0x7fe0a00b0a70 con 0x7fe0a00b4d60 2024-03-25T19:31:55.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.503+0000 7fe09e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:55.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.503+0000 7fe09e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:55.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.503+0000 7fe09e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:55.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.503+0000 7fe09e7fc640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:31:55.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.503+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe088003520 con 0x7fe0a00b4d60 2024-03-25T19:31:55.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.511+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125950000 0 0) 0x7fe0a00af940 con 0x7fe0a014be90 2024-03-25T19:31:55.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.511+0000 7fe09e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:55.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.511+0000 7fe09e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:55.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.511+0000 7fe09e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:55.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.511+0000 7fe09e7fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:31:55.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.511+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a00b0a70 con 0x7fe0a014be90 2024-03-25T19:31:55.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121329629 0 0) 0x7fe088003520 con 0x7fe0a00b4d60 2024-03-25T19:31:55.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 >> v1:172.21.15.134:6790/0 conn(0x7fe0a00b1730 legacy=0x7fe0a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:55.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 >> v1:172.21.15.134:6789/0 conn(0x7fe0a014be90 legacy=0x7fe0a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:55.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:55.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a00b1030 con 0x7fe0a00b4d60 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe090001690 con 0x7fe0a00b4d60 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe09e7fc640 10 monclient: dump: 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:55.517 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe0a6206640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.515+0000 7fe0a6206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0900017f0 con 0x7fe0a00b4d60 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe09e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe09e7fc640 1 -- 172.21.15.134:0/3901288799 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe090001c00 con 0x7fe0a00b4d60 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe0a6206640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe0a6206640 4 set_mon_vals no callback set 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe09e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe09e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.527+0000 7fe09e7fc640 10 monclient: dump: 2024-03-25T19:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:55.528 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.528+0000 7fe0a6206640 10 monclient: shutdown 2024-03-25T19:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.528+0000 7fe0a6206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.528+0000 7fe0a6206640 1 -- 172.21.15.134:0/3901288799 >> v1:172.21.15.159:6789/0 conn(0x7fe0a00b4d60 legacy=0x7fe0a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.528+0000 7fe0a6206640 1 -- 172.21.15.134:0/3901288799 shutdown_connections 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.528+0000 7fe0a6206640 1 -- 172.21.15.134:0/3901288799 >> 172.21.15.134:0/3901288799 conn(0x7fe0a00617a0 msgr2=0x7fe0a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.528+0000 7fe0a6206640 1 -- 172.21.15.134:0/3901288799 shutdown_connections 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.528+0000 7fe0a6206640 1 -- 172.21.15.134:0/3901288799 wait complete. 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 10 monclient: build_initial_monmap 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 10 monclient: monmap: 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:31:55.530 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 1 librados: starting msgr at 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 1 librados: starting objecter 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 1 Processor -- start 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 1 -- start start 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 1 librados: setting wanted keys 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 1 librados: calling monclient init 2024-03-25T19:31:55.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.529+0000 7fe0a6206640 10 monclient: init 2024-03-25T19:31:55.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.530+0000 7fe0a6206640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:55.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.530+0000 7fe0a6206640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:55.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.530+0000 7fe0a6206640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a00b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:31:55.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.530+0000 7fe0a6206640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a00b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:31:55.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.530+0000 7fe0a6206640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a014be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:31:55.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.531+0000 7fe0a6206640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a01440c0 con 0x7fe0a00b1730 2024-03-25T19:31:55.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.531+0000 7fe0a6206640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a013f020 con 0x7fe0a00b4d60 2024-03-25T19:31:55.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.531+0000 7fe0a6206640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0136d30 con 0x7fe0a014be90 2024-03-25T19:31:55.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.531+0000 7fe0a6206640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:55.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.531+0000 7fe0a6206640 10 monclient(hunting): authenticate will time out at 6147.689708s 2024-03-25T19:31:55.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.541+0000 7fe09ffff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe0a014be90 0x7fe0a01d8260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:39522/0 (socket says 172.21.15.134:39522) 2024-03-25T19:31:55.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.541+0000 7fe09ffff640 1 -- 172.21.15.134:0/1454819803 learned_addr learned my addr 172.21.15.134:0/1454819803 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:31:55.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.541+0000 7fe087fff640 10 client.?.objecter ms_handle_connect 0x7fe0a014be90 2024-03-25T19:31:55.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.541+0000 7fe087fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.542+0000 7fe087fff640 10 client.?.objecter ms_handle_connect 0x7fe0a00b1730 2024-03-25T19:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.542+0000 7fe087fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.544+0000 7fe087fff640 10 client.?.objecter ms_handle_connect 0x7fe0a00b4d60 2024-03-25T19:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.544+0000 7fe087fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:55.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455640484 0 0) 0x7fe0a013f020 con 0x7fe0a00b4d60 2024-03-25T19:31:55.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:55.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:55.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:55.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:31:55.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c004eb0 con 0x7fe0a00b4d60 2024-03-25T19:31:55.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318775812 0 0) 0x7fe0a01440c0 con 0x7fe0a00b1730 2024-03-25T19:31:55.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:55.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:55.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:55.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:31:55.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.552+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a013f020 con 0x7fe0a00b1730 2024-03-25T19:31:55.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.555+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568983483 0 0) 0x7fe0a0136d30 con 0x7fe0a014be90 2024-03-25T19:31:55.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.555+0000 7fe087fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:55.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.555+0000 7fe087fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:55.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.555+0000 7fe087fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:55.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.555+0000 7fe087fff640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:31:55.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.555+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a01440c0 con 0x7fe0a014be90 2024-03-25T19:31:55.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.571+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 569576044 0 0) 0x7fe08c004eb0 con 0x7fe0a00b4d60 2024-03-25T19:31:55.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.571+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a0136d30 con 0x7fe0a00b4d60 2024-03-25T19:31:55.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.571+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe0940018c0 con 0x7fe0a00b4d60 2024-03-25T19:31:55.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.571+0000 7fe087fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:55.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.571+0000 7fe087fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.571+0000 7fe087fff640 10 monclient(hunting): dump: 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.571+0000 7fe087fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.576+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 341608055 0 0) 0x7fe0a013f020 con 0x7fe0a00b1730 2024-03-25T19:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.576+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe08c004eb0 con 0x7fe0a00b1730 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.576+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe090001b00 con 0x7fe0a00b1730 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.576+0000 7fe087fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.577+0000 7fe087fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.577+0000 7fe087fff640 10 monclient(hunting): dump: 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:55.577 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.580+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345848730 0 0) 0x7fe0a01440c0 con 0x7fe0a014be90 2024-03-25T19:31:55.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.580+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a013f020 con 0x7fe0a014be90 2024-03-25T19:31:55.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.580+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe098001c60 con 0x7fe0a014be90 2024-03-25T19:31:55.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.580+0000 7fe087fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:55.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.580+0000 7fe087fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:55.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.580+0000 7fe087fff640 10 monclient(hunting): dump: 2024-03-25T19:31:55.581 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:55.581 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:55.581 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.582 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.582 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:55.582 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:55.582 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:55.582 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:55.582 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:55.582 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.588+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2630140311 0 0) 0x7fe0a013f020 con 0x7fe0a014be90 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.588+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 >> v1:172.21.15.159:6789/0 conn(0x7fe0a00b4d60 legacy=0x7fe0a01d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.588+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 >> v1:172.21.15.134:6789/0 conn(0x7fe0a00b1730 legacy=0x7fe0a01439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.588+0000 7fe087fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.588+0000 7fe087fff640 1 monclient: found mon.c 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.588+0000 7fe087fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe087fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a0136ec0 con 0x7fe0a014be90 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe087fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe087fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe087fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 5 monclient: authenticate success, global_id 4367 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 10 monclient: _renew_subs 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a01e33d0 con 0x7fe0a014be90 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 10 monclient: _renew_subs 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a01e3740 con 0x7fe0a014be90 2024-03-25T19:31:55.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.589+0000 7fe0a6206640 1 librados: init done 2024-03-25T19:31:55.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.590+0000 7fe0a6206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:55.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.590+0000 7fe0a6206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:55.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.590+0000 7fe0a6206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:55.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.590+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a01d79a0 con 0x7fe0a014be90 2024-03-25T19:31:55.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.882+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe098003990 con 0x7fe0a014be90 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.882+0000 7fe087fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.882+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe098003da0 con 0x7fe0a014be90 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.882+0000 7fe087fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.882+0000 7fe087fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.882+0000 7fe087fff640 10 monclient: dump: 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:31:55.883 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:55.884 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:31:55.884 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:31:55.884 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:31:55.884 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:31:55.884 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:31:55.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.882+0000 7fe09dffb640 4 set_mon_vals no callback set 2024-03-25T19:31:55.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.883+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fe098003fc0 con 0x7fe0a014be90 2024-03-25T19:31:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.884+0000 7fe09f7fe640 10 monclient: get_auth_request con 0x7fe08c06e160 auth_method 0 2024-03-25T19:31:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.884+0000 7fe087fff640 10 client.4367.objecter ms_handle_connect 0x7fe08c06e160 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fe098054990 con 0x7fe0a014be90 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 10 client.4367.objecter ms_dispatch 0x7fe0a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0980504a0 con 0x7fe0a014be90 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:55.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:55.892+0000 7fe087fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:56.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.147+0000 7fe0a6206640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:56.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.147+0000 7fe0a6206640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:56.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.147+0000 7fe0a6206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:31:56.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.147+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0a01401f0 con 0x7fe0a014be90 2024-03-25T19:31:56.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.182+0000 7fe087fff640 1 -- 172.21.15.134:0/1454819803 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fe0980523d0 con 0x7fe0a014be90 2024-03-25T19:31:56.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.182+0000 7fe087fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:56.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.182+0000 7fe087fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:56.182 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:31:56.182 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:56.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 10 librados: watch_flush enter 2024-03-25T19:31:56.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 10 librados: watch_flush exit 2024-03-25T19:31:56.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 10 client.4367.objecter successfully canceled tick 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 >> v1:172.21.15.159:6800/1875874887 conn(0x7fe08c06e160 legacy=0x7fe08c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 10 monclient: shutdown 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 >> v1:172.21.15.134:6790/0 conn(0x7fe0a014be90 legacy=0x7fe0a01d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 shutdown_connections 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 >> 172.21.15.134:0/1454819803 conn(0x7fe0a00617a0 msgr2=0x7fe0a013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.185+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 shutdown_connections 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.186+0000 7fe0a6206640 1 -- 172.21.15.134:0/1454819803 wait complete. 2024-03-25T19:31:56.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:31:56.186+0000 7fe0a6206640 1 librados: shutdown 2024-03-25T19:31:56.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:00.105 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:04.777 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.119 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:04.791 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.163 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:04.835 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.184 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:04.856 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.187 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:04.834 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.194 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:04.841 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.195 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:04.868 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.222 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:04.894 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:00.226 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:04.873 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:03.004 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.117+0000 7f24e1886640 10 monclient: get_monmap_and_config 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.117+0000 7f24e1886640 10 monclient: build_initial_monmap 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.117+0000 7f24e1886640 10 monclient: monmap: 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:03.118820+0000 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:03.118820+0000 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:03.118 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:03.119 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:03.119 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.118+0000 7f24e1886640 1 Processor -- start 2024-03-25T19:32:03.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.118+0000 7f24e1886640 1 -- start start 2024-03-25T19:32:03.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.118+0000 7f24e1886640 10 monclient: init 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 10 monclient(hunting): picked mon.noname-a con 0x7f24dc0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 10 monclient(hunting): picked mon.noname-c con 0x7f24dc13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 10 monclient(hunting): picked mon.noname-b con 0x7f24dc1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc142520 con 0x7f24dc0a98f0 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc0b28a0 con 0x7f24dc13f710 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc0b2a30 con 0x7f24dc1430c0 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:03.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.119+0000 7f24e1886640 10 monclient(hunting): authenticate will time out at 6155.278205s 2024-03-25T19:32:03.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.128+0000 7f24db7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f24dc1430c0 0x7f24dc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59150/0 (socket says 172.21.15.134:59150) 2024-03-25T19:32:03.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.128+0000 7f24db7fe640 1 -- 172.21.15.134:0/1421431706 learned_addr learned my addr 172.21.15.134:0/1421431706 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.143+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365394536 0 0) 0x7f24dc0b28a0 con 0x7f24dc13f710 2024-03-25T19:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.143+0000 7f24d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.143+0000 7f24d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.143+0000 7f24d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.143+0000 7f24d9ffb640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.143+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c4003520 con 0x7f24dc13f710 2024-03-25T19:32:03.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.152+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765023246 0 0) 0x7f24dc142520 con 0x7f24dc0a98f0 2024-03-25T19:32:03.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.152+0000 7f24d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:03.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.152+0000 7f24d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:03.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.152+0000 7f24d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:03.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.152+0000 7f24d9ffb640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:32:03.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.152+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24dc0b28a0 con 0x7f24dc0a98f0 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368346457 0 0) 0x7f24c4003520 con 0x7f24dc13f710 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 >> v1:172.21.15.134:6790/0 conn(0x7f24dc1430c0 legacy=0x7f24dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 >> v1:172.21.15.134:6789/0 conn(0x7f24dc0a98f0 legacy=0x7f24dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 1 monclient: found mon.noname-c 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc0b44c0 con 0x7f24dc13f710 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:03.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f24d0001690 con 0x7f24dc13f710 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.156+0000 7f24d9ffb640 10 monclient: dump: 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.157+0000 7f24e1886640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:32:03.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.157+0000 7f24e1886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:03.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.168+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24d00017f0 con 0x7f24dc13f710 2024-03-25T19:32:03.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:03.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24d9ffb640 1 -- 172.21.15.134:0/1421431706 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f24d0001c00 con 0x7f24dc13f710 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24e1886640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24e1886640 4 set_mon_vals no callback set 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24d9ffb640 10 monclient: dump: 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:03.169 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24e1886640 10 monclient: shutdown 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24e1886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24e1886640 1 -- 172.21.15.134:0/1421431706 >> v1:172.21.15.159:6789/0 conn(0x7f24dc13f710 legacy=0x7f24dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24e1886640 1 -- 172.21.15.134:0/1421431706 shutdown_connections 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.169+0000 7f24e1886640 1 -- 172.21.15.134:0/1421431706 >> 172.21.15.134:0/1421431706 conn(0x7f24dc0b7ea0 msgr2=0x7f24dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.170+0000 7f24e1886640 1 -- 172.21.15.134:0/1421431706 shutdown_connections 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.170+0000 7f24e1886640 1 -- 172.21.15.134:0/1421431706 wait complete. 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.170+0000 7f24e1886640 10 monclient: build_initial_monmap 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.170+0000 7f24e1886640 10 monclient: monmap: 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:03.171 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.171+0000 7f24e1886640 1 librados: starting msgr at 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.171+0000 7f24e1886640 1 librados: starting objecter 2024-03-25T19:32:03.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.171+0000 7f24e1886640 1 Processor -- start 2024-03-25T19:32:03.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.171+0000 7f24e1886640 1 -- start start 2024-03-25T19:32:03.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.171+0000 7f24e1886640 1 librados: setting wanted keys 2024-03-25T19:32:03.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.171+0000 7f24e1886640 1 librados: calling monclient init 2024-03-25T19:32:03.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.171+0000 7f24e1886640 10 monclient: init 2024-03-25T19:32:03.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:03.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:03.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 10 monclient(hunting): picked mon.noname-a con 0x7f24dc0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:03.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 10 monclient(hunting): picked mon.noname-b con 0x7f24dc13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:03.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 10 monclient(hunting): picked mon.noname-c con 0x7f24dc1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:03.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc138a50 con 0x7f24dc0a98f0 2024-03-25T19:32:03.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc0ab9e0 con 0x7f24dc13f710 2024-03-25T19:32:03.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc1d7880 con 0x7f24dc1430c0 2024-03-25T19:32:03.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:03.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.172+0000 7f24e1886640 10 monclient(hunting): authenticate will time out at 6155.331394s 2024-03-25T19:32:03.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.183+0000 7f24db7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f24dc1430c0 0x7f24dc1e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59180/0 (socket says 172.21.15.134:59180) 2024-03-25T19:32:03.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.183+0000 7f24db7fe640 1 -- 172.21.15.134:0/885491548 learned_addr learned my addr 172.21.15.134:0/885491548 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:03.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.185+0000 7f24c3fff640 10 client.?.objecter ms_handle_connect 0x7f24dc1430c0 2024-03-25T19:32:03.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.185+0000 7f24c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:03.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.186+0000 7f24c3fff640 10 client.?.objecter ms_handle_connect 0x7f24dc0a98f0 2024-03-25T19:32:03.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.186+0000 7f24c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:03.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.188+0000 7f24c3fff640 10 client.?.objecter ms_handle_connect 0x7f24dc13f710 2024-03-25T19:32:03.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.188+0000 7f24c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:03.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.196+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315068504 0 0) 0x7f24dc0ab9e0 con 0x7f24dc13f710 2024-03-25T19:32:03.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.196+0000 7f24c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:03.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.196+0000 7f24c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:03.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.196+0000 7f24c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:03.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.196+0000 7f24c3fff640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:32:03.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.196+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c8004eb0 con 0x7f24dc13f710 2024-03-25T19:32:03.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.202+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327216842 0 0) 0x7f24dc138a50 con 0x7f24dc0a98f0 2024-03-25T19:32:03.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.202+0000 7f24c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:03.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.202+0000 7f24c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:03.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.202+0000 7f24c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:03.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.202+0000 7f24c3fff640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:32:03.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.202+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24dc0ab9e0 con 0x7f24dc0a98f0 2024-03-25T19:32:03.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.205+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091426675 0 0) 0x7f24dc1d7880 con 0x7f24dc1430c0 2024-03-25T19:32:03.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.205+0000 7f24c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:03.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.205+0000 7f24c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:03.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.205+0000 7f24c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:03.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.205+0000 7f24c3fff640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:32:03.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.205+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24dc138a50 con 0x7f24dc1430c0 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.218+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2358913774 0 0) 0x7f24c8004eb0 con 0x7f24dc13f710 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.218+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24dc1d7880 con 0x7f24dc13f710 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.218+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f24d0001b00 con 0x7f24dc13f710 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.218+0000 7f24c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.218+0000 7f24c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.218+0000 7f24c3fff640 10 monclient(hunting): dump: 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.218 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:03.219 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:03.219 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:03.219 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:03.219 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:03.219 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.218+0000 7f24c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:03.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.222+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903335774 0 0) 0x7f24dc138a50 con 0x7f24dc1430c0 2024-03-25T19:32:03.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.222+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c8004eb0 con 0x7f24dc1430c0 2024-03-25T19:32:03.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.222+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f24d4001b10 con 0x7f24dc1430c0 2024-03-25T19:32:03.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.222+0000 7f24c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:03.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.222+0000 7f24c3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.222+0000 7f24c3fff640 10 monclient(hunting): dump: 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:03.223 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.228+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1842901373 0 0) 0x7f24dc0ab9e0 con 0x7f24dc0a98f0 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.228+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24dc138a50 con 0x7f24dc0a98f0 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.228+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f24cc0019f0 con 0x7f24dc0a98f0 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.228+0000 7f24c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.228+0000 7f24c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.228+0000 7f24c3fff640 10 monclient(hunting): dump: 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:03.228 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3180515578 0 0) 0x7f24c8004eb0 con 0x7f24dc1430c0 2024-03-25T19:32:03.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 >> v1:172.21.15.159:6789/0 conn(0x7f24dc13f710 legacy=0x7f24dc1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:03.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 >> v1:172.21.15.134:6789/0 conn(0x7f24dc0a98f0 legacy=0x7f24dc138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:03.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:03.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 1 monclient: found mon.c 2024-03-25T19:32:03.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc1d84b0 con 0x7f24dc1430c0 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 5 monclient: authenticate success, global_id 4373 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 10 monclient: _renew_subs 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24dc1d51f0 con 0x7f24dc1430c0 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 10 monclient: _renew_subs 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24dc138e60 con 0x7f24dc1430c0 2024-03-25T19:32:03.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.233+0000 7f24e1886640 1 librados: init done 2024-03-25T19:32:03.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.234+0000 7f24e1886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:03.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.234+0000 7f24e1886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:03.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.234+0000 7f24e1886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:03.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.234+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24dc1d76e0 con 0x7f24dc1430c0 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24d4002000 con 0x7f24dc1430c0 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f24d4004b60 con 0x7f24dc1430c0 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24c3fff640 10 monclient: dump: 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:03.530 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24d97fa640 4 set_mon_vals no callback set 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.530+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f24d401ab30 con 0x7f24dc1430c0 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.531+0000 7f24daffd640 10 monclient: get_auth_request con 0x7f24c806e0c0 auth_method 0 2024-03-25T19:32:03.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.532+0000 7f24c3fff640 10 client.4373.objecter ms_handle_connect 0x7f24c806e0c0 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f24d4054800 con 0x7f24dc1430c0 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 10 client.4373.objecter ms_dispatch 0x7f24dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24d4050310 con 0x7f24dc1430c0 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.538+0000 7f24c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:03.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.768+0000 7f24e1886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:03.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.768+0000 7f24e1886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:03.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.768+0000 7f24e1886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:03.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.768+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24dc1d7a10 con 0x7f24dc1430c0 2024-03-25T19:32:03.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.802+0000 7f24c3fff640 1 -- 172.21.15.134:0/885491548 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f24d4052240 con 0x7f24dc1430c0 2024-03-25T19:32:03.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.802+0000 7f24c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:03.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.802+0000 7f24c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:03.803 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:03.803 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.804+0000 7f24e1886640 10 librados: watch_flush enter 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.804+0000 7f24e1886640 10 librados: watch_flush exit 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.804+0000 7f24e1886640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.804+0000 7f24e1886640 10 client.4373.objecter successfully canceled tick 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.804+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 >> v1:172.21.15.159:6800/1875874887 conn(0x7f24c806e0c0 legacy=0x7f24c8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 10 monclient: shutdown 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:03.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 >> v1:172.21.15.134:6790/0 conn(0x7f24dc1430c0 legacy=0x7f24dc1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:03.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 shutdown_connections 2024-03-25T19:32:03.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 >> 172.21.15.134:0/885491548 conn(0x7f24dc0b7ea0 msgr2=0x7f24dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:03.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 shutdown_connections 2024-03-25T19:32:03.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 1 -- 172.21.15.134:0/885491548 wait complete. 2024-03-25T19:32:03.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:03.805+0000 7f24e1886640 1 librados: shutdown 2024-03-25T19:32:03.816 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:10.622 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:10.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.740+0000 7f27efdcf640 10 monclient: get_monmap_and_config 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.740+0000 7f27efdcf640 10 monclient: build_initial_monmap 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.740+0000 7f27efdcf640 10 monclient: monmap: 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:10.741920+0000 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:10.741920+0000 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:10.741 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.741+0000 7f27efdcf640 1 Processor -- start 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.741+0000 7f27efdcf640 1 -- start start 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.741+0000 7f27efdcf640 10 monclient: init 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 10 monclient(hunting): picked mon.noname-a con 0x7f27e80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 10 monclient(hunting): picked mon.noname-c con 0x7f27e813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 10 monclient(hunting): picked mon.noname-b con 0x7f27e81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e8142520 con 0x7f27e80a98f0 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e80b28a0 con 0x7f27e813f710 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e80b2a30 con 0x7f27e81430c0 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:10.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.742+0000 7f27efdcf640 10 monclient(hunting): authenticate will time out at 6162.901038s 2024-03-25T19:32:10.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.756+0000 7f27ed36a640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f27e813f710 0x7f27e81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:33218/0 (socket says 172.21.15.134:33218) 2024-03-25T19:32:10.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.756+0000 7f27ed36a640 1 -- 172.21.15.134:0/2970377972 learned_addr learned my addr 172.21.15.134:0/2970377972 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:10.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.770+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703588336 0 0) 0x7f27e80b28a0 con 0x7f27e813f710 2024-03-25T19:32:10.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.770+0000 7f27ecb69640 10 monclient(hunting): _negotiate 2024-03-25T19:32:10.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.770+0000 7f27ecb69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:10.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.770+0000 7f27ecb69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:10.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.770+0000 7f27ecb69640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:32:10.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.770+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d4003520 con 0x7f27e813f710 2024-03-25T19:32:10.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.775+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758154538 0 0) 0x7f27e8142520 con 0x7f27e80a98f0 2024-03-25T19:32:10.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.775+0000 7f27ecb69640 10 monclient(hunting): _negotiate 2024-03-25T19:32:10.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.775+0000 7f27ecb69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:10.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.775+0000 7f27ecb69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:10.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.775+0000 7f27ecb69640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:32:10.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.775+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e80b28a0 con 0x7f27e80a98f0 2024-03-25T19:32:10.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.778+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252085266 0 0) 0x7f27e80b2a30 con 0x7f27e81430c0 2024-03-25T19:32:10.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.778+0000 7f27ecb69640 10 monclient(hunting): _negotiate 2024-03-25T19:32:10.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.778+0000 7f27ecb69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:10.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.778+0000 7f27ecb69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:10.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.778+0000 7f27ecb69640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:32:10.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.778+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e8142520 con 0x7f27e81430c0 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3331924267 0 0) 0x7f27e80b28a0 con 0x7f27e80a98f0 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 >> v1:172.21.15.134:6790/0 conn(0x7f27e81430c0 legacy=0x7f27e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 >> v1:172.21.15.159:6789/0 conn(0x7f27e813f710 legacy=0x7f27e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 10 monclient: _finish_hunting 0 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 1 monclient: found mon.noname-a 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e80b44c0 con 0x7f27e80a98f0 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 10 monclient: _finish_auth 0 2024-03-25T19:32:10.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 10 monclient: _check_auth_tickets 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f27e4001670 con 0x7f27e80a98f0 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27efdcf640 5 monclient: authenticate success, global_id 4410 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27ecb69640 10 monclient: dump: 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:10.792 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:10.793 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.791+0000 7f27efdcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27e40017d0 con 0x7f27e80a98f0 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27ecb69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27ecb69640 1 -- 172.21.15.134:0/2970377972 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f27e4001be0 con 0x7f27e80a98f0 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27ecb69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27ecb69640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27ecb69640 10 monclient: dump: 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.804 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:10.805 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:10.805 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:10.805 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:10.805 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:10.805 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27efdcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:10.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.804+0000 7f27efdcf640 4 set_mon_vals no callback set 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.805+0000 7f27efdcf640 10 monclient: shutdown 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.805+0000 7f27efdcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.805+0000 7f27efdcf640 1 -- 172.21.15.134:0/2970377972 >> v1:172.21.15.134:6789/0 conn(0x7f27e80a98f0 legacy=0x7f27e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.805+0000 7f27efdcf640 1 -- 172.21.15.134:0/2970377972 shutdown_connections 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.805+0000 7f27efdcf640 1 -- 172.21.15.134:0/2970377972 >> 172.21.15.134:0/2970377972 conn(0x7f27e80b7ea0 msgr2=0x7f27e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.805+0000 7f27efdcf640 1 -- 172.21.15.134:0/2970377972 shutdown_connections 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.805+0000 7f27efdcf640 1 -- 172.21.15.134:0/2970377972 wait complete. 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.806+0000 7f27efdcf640 10 monclient: build_initial_monmap 2024-03-25T19:32:10.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.806+0000 7f27efdcf640 10 monclient: monmap: 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.806+0000 7f27efdcf640 1 librados: starting msgr at 2024-03-25T19:32:10.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.806+0000 7f27efdcf640 1 librados: starting objecter 2024-03-25T19:32:10.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 1 Processor -- start 2024-03-25T19:32:10.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 1 -- start start 2024-03-25T19:32:10.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 1 librados: setting wanted keys 2024-03-25T19:32:10.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 1 librados: calling monclient init 2024-03-25T19:32:10.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient: init 2024-03-25T19:32:10.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient(hunting): picked mon.noname-b con 0x7f27e80a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient(hunting): picked mon.noname-c con 0x7f27e813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient(hunting): picked mon.noname-a con 0x7f27e81430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e8139a30 con 0x7f27e81430c0 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e80706e0 con 0x7f27e80a98f0 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e81dfc90 con 0x7f27e813f710 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:10.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.807+0000 7f27efdcf640 10 monclient(hunting): authenticate will time out at 6162.966590s 2024-03-25T19:32:10.817 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.817+0000 7f27edb6b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f27e80a98f0 0x7f27e80716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:33258/0 (socket says 172.21.15.134:33258) 2024-03-25T19:32:10.817 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.817+0000 7f27edb6b640 1 -- 172.21.15.134:0/1210256860 learned_addr learned my addr 172.21.15.134:0/1210256860 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:10.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.818+0000 7f27de7fc640 10 client.?.objecter ms_handle_connect 0x7f27e80a98f0 2024-03-25T19:32:10.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.818+0000 7f27de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:10.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.827+0000 7f27de7fc640 10 client.?.objecter ms_handle_connect 0x7f27e813f710 2024-03-25T19:32:10.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.827+0000 7f27de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:10.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.832+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332642879 0 0) 0x7f27e80706e0 con 0x7f27e80a98f0 2024-03-25T19:32:10.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.832+0000 7f27de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:10.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.832+0000 7f27de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:10.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.832+0000 7f27de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:10.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.832+0000 7f27de7fc640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:32:10.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.832+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc004eb0 con 0x7f27e80a98f0 2024-03-25T19:32:10.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.835+0000 7f27de7fc640 10 client.?.objecter ms_handle_connect 0x7f27e81430c0 2024-03-25T19:32:10.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.835+0000 7f27de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:10.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.836+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719407683 0 0) 0x7f27e81dfc90 con 0x7f27e813f710 2024-03-25T19:32:10.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.836+0000 7f27de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:10.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.836+0000 7f27de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:10.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.836+0000 7f27de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:10.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.836+0000 7f27de7fc640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:32:10.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.836+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e80706e0 con 0x7f27e813f710 2024-03-25T19:32:10.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.853+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516487000 0 0) 0x7f27e8139a30 con 0x7f27e81430c0 2024-03-25T19:32:10.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.854+0000 7f27de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:10.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.854+0000 7f27de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:10.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.854+0000 7f27de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:10.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.854+0000 7f27de7fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:32:10.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.854+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e81dfc90 con 0x7f27e81430c0 2024-03-25T19:32:10.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.865+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 629504259 0 0) 0x7f27cc004eb0 con 0x7f27e80a98f0 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.865+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27e8139a30 con 0x7f27e80a98f0 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.865+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f27e4001670 con 0x7f27e80a98f0 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.865+0000 7f27de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.865+0000 7f27de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.865+0000 7f27de7fc640 10 monclient(hunting): dump: 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:10.866 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:10.867 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:10.867 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.865+0000 7f27de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.868+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1767697760 0 0) 0x7f27e80706e0 con 0x7f27e813f710 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.868+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27cc004eb0 con 0x7f27e813f710 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.868+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f27d8001950 con 0x7f27e813f710 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.868+0000 7f27de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.868+0000 7f27de7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.868+0000 7f27de7fc640 10 monclient(hunting): dump: 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.868 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:10.869 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:10.869 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:10.869 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:10.869 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:10.869 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.869+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154822664 0 0) 0x7f27e81dfc90 con 0x7f27e81430c0 2024-03-25T19:32:10.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.869+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27e80706e0 con 0x7f27e81430c0 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.869+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f27e0001bd0 con 0x7f27e81430c0 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.869+0000 7f27de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.869+0000 7f27de7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.869+0000 7f27de7fc640 10 monclient(hunting): dump: 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:10.870 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 580412693 0 0) 0x7f27e80706e0 con 0x7f27e81430c0 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 >> v1:172.21.15.134:6790/0 conn(0x7f27e813f710 legacy=0x7f27e8138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 >> v1:172.21.15.159:6789/0 conn(0x7f27e80a98f0 legacy=0x7f27e80716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 1 monclient: found mon.a 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e81dfe20 con 0x7f27e81430c0 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27efdcf640 5 monclient: authenticate success, global_id 4413 2024-03-25T19:32:10.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27efdcf640 10 monclient: _renew_subs 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27efdcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.878+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e8070be0 con 0x7f27e81430c0 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.879+0000 7f27efdcf640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.879+0000 7f27efdcf640 10 monclient: _renew_subs 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.879+0000 7f27efdcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.879+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e81df250 con 0x7f27e81430c0 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.879+0000 7f27efdcf640 1 librados: init done 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.880+0000 7f27efdcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:10.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.880+0000 7f27efdcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:10.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.880+0000 7f27efdcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:10.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.880+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e8072120 con 0x7f27e81430c0 2024-03-25T19:32:10.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.905+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27e0001fe0 con 0x7f27e81430c0 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.905+0000 7f27de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.905+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f27e0003b30 con 0x7f27e81430c0 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.905+0000 7f27de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.905+0000 7f27de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.905+0000 7f27de7fc640 10 monclient: dump: 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:10.906 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:10.907 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:10.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.905+0000 7f27dffff640 4 set_mon_vals no callback set 2024-03-25T19:32:10.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.906+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f27e0019b00 con 0x7f27e81430c0 2024-03-25T19:32:10.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.907+0000 7f27edb6b640 10 monclient: get_auth_request con 0x7f27cc06e160 auth_method 0 2024-03-25T19:32:10.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.907+0000 7f27de7fc640 10 client.4413.objecter ms_handle_connect 0x7f27cc06e160 2024-03-25T19:32:10.910 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.910+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f27e00537e0 con 0x7f27e81430c0 2024-03-25T19:32:10.910 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.910+0000 7f27de7fc640 10 client.4413.objecter ms_dispatch 0x7f27e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:10.910 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.910+0000 7f27de7fc640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:10.910 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.910+0000 7f27de7fc640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:10.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:10.910+0000 7f27de7fc640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-25T19:32:11.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.197+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27e004f120 con 0x7f27e81430c0 2024-03-25T19:32:11.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.197+0000 7f27de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:11.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.197+0000 7f27de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:11.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.421+0000 7f27efdcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:11.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.421+0000 7f27efdcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:11.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.421+0000 7f27efdcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:11.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.421+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27e80704e0 con 0x7f27e81430c0 2024-03-25T19:32:11.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.453+0000 7f27de7fc640 1 -- 172.21.15.134:0/1210256860 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f27e00013f0 con 0x7f27e81430c0 2024-03-25T19:32:11.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.454+0000 7f27de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.454+0000 7f27de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:11.454 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:11.454 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:11.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 10 librados: watch_flush enter 2024-03-25T19:32:11.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 10 librados: watch_flush exit 2024-03-25T19:32:11.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-25T19:32:11.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 10 client.4413.objecter successfully canceled tick 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 >> v1:172.21.15.159:6800/1875874887 conn(0x7f27cc06e160 legacy=0x7f27cc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 10 monclient: shutdown 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.456+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 >> v1:172.21.15.134:6789/0 conn(0x7f27e81430c0 legacy=0x7f27e8139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.457+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 shutdown_connections 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.457+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 >> 172.21.15.134:0/1210256860 conn(0x7f27e80b7ea0 msgr2=0x7f27e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.457+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 shutdown_connections 2024-03-25T19:32:11.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.457+0000 7f27efdcf640 1 -- 172.21.15.134:0/1210256860 wait complete. 2024-03-25T19:32:11.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:11.457+0000 7f27efdcf640 1 librados: shutdown 2024-03-25T19:32:11.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:18.276 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:18.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.375+0000 7f406d8a8640 10 monclient: get_monmap_and_config 2024-03-25T19:32:18.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.375+0000 7f406d8a8640 10 monclient: build_initial_monmap 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.375+0000 7f406d8a8640 10 monclient: monmap: 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:18.376852+0000 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:18.376852+0000 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:18.376 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.376+0000 7f406d8a8640 1 Processor -- start 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.376+0000 7f406d8a8640 1 -- start start 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.376+0000 7f406d8a8640 10 monclient: init 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f40680a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f406813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f40681430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4068142520 con 0x7f40680a98f0 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40680b28a0 con 0x7f40681430c0 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40680b2a30 con 0x7f406813f710 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:18.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.377+0000 7f406d8a8640 10 monclient(hunting): authenticate will time out at 6170.535957s 2024-03-25T19:32:18.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.385+0000 7f4066ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f40680a98f0 0x7f40680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:48036/0 (socket says 172.21.15.134:48036) 2024-03-25T19:32:18.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.385+0000 7f4066ffd640 1 -- 172.21.15.134:0/2694358945 learned_addr learned my addr 172.21.15.134:0/2694358945 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:18.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.395+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487452374 0 0) 0x7f4068142520 con 0x7f40680a98f0 2024-03-25T19:32:18.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.395+0000 7f40667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:18.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.395+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:18.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.395+0000 7f40667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:18.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.395+0000 7f40667fc640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:32:18.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.395+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4054003520 con 0x7f40680a98f0 2024-03-25T19:32:18.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480799564 0 0) 0x7f4054003520 con 0x7f40680a98f0 2024-03-25T19:32:18.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 >> v1:172.21.15.134:6790/0 conn(0x7f406813f710 legacy=0x7f40681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:32:18.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 >> v1:172.21.15.159:6789/0 conn(0x7f40681430c0 legacy=0x7f406814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 1 monclient: found mon.noname-a 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40680b44c0 con 0x7f40680a98f0 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4058001690 con 0x7f40680a98f0 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f40667fc640 10 monclient: dump: 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:18.408 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f406d8a8640 5 monclient: authenticate success, global_id 4416 2024-03-25T19:32:18.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.407+0000 7f406d8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4058001860 con 0x7f40680a98f0 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f40667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f40667fc640 1 -- 172.21.15.134:0/2694358945 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4058001cb0 con 0x7f40680a98f0 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f40667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f40667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f40667fc640 10 monclient: dump: 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.423 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:18.424 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:18.424 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:18.424 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:18.424 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:18.424 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:18.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f406d8a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:18.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.423+0000 7f406d8a8640 4 set_mon_vals no callback set 2024-03-25T19:32:18.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.424+0000 7f406d8a8640 10 monclient: shutdown 2024-03-25T19:32:18.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.424+0000 7f406d8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.424+0000 7f406d8a8640 1 -- 172.21.15.134:0/2694358945 >> v1:172.21.15.134:6789/0 conn(0x7f40680a98f0 legacy=0x7f40680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.424+0000 7f406d8a8640 1 -- 172.21.15.134:0/2694358945 shutdown_connections 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.424+0000 7f406d8a8640 1 -- 172.21.15.134:0/2694358945 >> 172.21.15.134:0/2694358945 conn(0x7f40680b7ea0 msgr2=0x7f40680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.424+0000 7f406d8a8640 1 -- 172.21.15.134:0/2694358945 shutdown_connections 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.424+0000 7f406d8a8640 1 -- 172.21.15.134:0/2694358945 wait complete. 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 10 monclient: build_initial_monmap 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 10 monclient: monmap: 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:18.427 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 1 librados: starting msgr at 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 1 librados: starting objecter 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 1 Processor -- start 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 1 -- start start 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 1 librados: setting wanted keys 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 1 librados: calling monclient init 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.425+0000 7f406d8a8640 10 monclient: init 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f40680a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:18.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f406813f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:18.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f40681430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:18.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40681d76e0 con 0x7f406813f710 2024-03-25T19:32:18.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40680abd80 con 0x7f40681430c0 2024-03-25T19:32:18.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4068137c90 con 0x7f40680a98f0 2024-03-25T19:32:18.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:18.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.426+0000 7f406d8a8640 10 monclient(hunting): authenticate will time out at 6170.585001s 2024-03-25T19:32:18.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.439+0000 7f40677fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f40681430c0 0x7f40681e2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:37914/0 (socket says 172.21.15.134:37914) 2024-03-25T19:32:18.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.439+0000 7f40677fe640 1 -- 172.21.15.134:0/2506801261 learned_addr learned my addr 172.21.15.134:0/2506801261 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:18.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.442+0000 7f4064ff9640 10 client.?.objecter ms_handle_connect 0x7f40681430c0 2024-03-25T19:32:18.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.442+0000 7f4064ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:18.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.447+0000 7f4064ff9640 10 client.?.objecter ms_handle_connect 0x7f40680a98f0 2024-03-25T19:32:18.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.447+0000 7f4064ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:18.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180431814 0 0) 0x7f40680abd80 con 0x7f40681430c0 2024-03-25T19:32:18.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:18.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:18.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:18.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:32:18.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4048004eb0 con 0x7f40681430c0 2024-03-25T19:32:18.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 10 client.?.objecter ms_handle_connect 0x7f406813f710 2024-03-25T19:32:18.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.449+0000 7f4064ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:18.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.455+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342675438 0 0) 0x7f4068137c90 con 0x7f40680a98f0 2024-03-25T19:32:18.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.455+0000 7f4064ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:18.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.455+0000 7f4064ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:18.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.455+0000 7f4064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:18.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.455+0000 7f4064ff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:32:18.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.455+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40680abd80 con 0x7f40680a98f0 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.461+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3515972010 0 0) 0x7f4048004eb0 con 0x7f40681430c0 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.461+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4068137c90 con 0x7f40681430c0 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.461+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4060001ad0 con 0x7f40681430c0 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.461+0000 7f4064ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.461+0000 7f4064ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.461+0000 7f4064ff9640 10 monclient(hunting): dump: 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:18.461 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:18.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.461+0000 7f4064ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:18.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1093185757 0 0) 0x7f40680abd80 con 0x7f40680a98f0 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4048004eb0 con 0x7f40680a98f0 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4058001b70 con 0x7f40680a98f0 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 10 monclient(hunting): dump: 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.466 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029139948 0 0) 0x7f40681d76e0 con 0x7f406813f710 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:18.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:18.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:32:18.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.466+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40680abd80 con 0x7f406813f710 2024-03-25T19:32:18.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 512476257 0 0) 0x7f4068137c90 con 0x7f40681430c0 2024-03-25T19:32:18.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 >> v1:172.21.15.134:6790/0 conn(0x7f40680a98f0 legacy=0x7f40681d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:18.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 >> v1:172.21.15.134:6789/0 conn(0x7f406813f710 legacy=0x7f4068138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 1 monclient: found mon.b 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40681d41e0 con 0x7f40681430c0 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 10 monclient: _finish_auth 0 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f4064ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 10 monclient: _renew_subs 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40681d82d0 con 0x7f40681430c0 2024-03-25T19:32:18.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:18.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 10 monclient: _renew_subs 2024-03-25T19:32:18.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:18.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4068138d80 con 0x7f40681430c0 2024-03-25T19:32:18.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.469+0000 7f406d8a8640 1 librados: init done 2024-03-25T19:32:18.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.470+0000 7f406d8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:18.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.470+0000 7f406d8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:18.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.470+0000 7f406d8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:18.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.470+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40680627b0 con 0x7f40681430c0 2024-03-25T19:32:18.491 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.492+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4060001fe0 con 0x7f40681430c0 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.492+0000 7f4064ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.492+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4060004d80 con 0x7f40681430c0 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.492+0000 7f4064ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.492+0000 7f4064ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.492+0000 7f4064ff9640 10 monclient: dump: 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:18.492 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:18.493 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:18.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.492+0000 7f4065ffb640 4 set_mon_vals no callback set 2024-03-25T19:32:18.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.493+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f406001ad50 con 0x7f40681430c0 2024-03-25T19:32:18.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.493+0000 7f4066ffd640 10 monclient: get_auth_request con 0x7f404806def0 auth_method 0 2024-03-25T19:32:18.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.493+0000 7f4064ff9640 10 client.4378.objecter ms_handle_connect 0x7f404806def0 2024-03-25T19:32:18.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.494+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f4060054a60 con 0x7f40681430c0 2024-03-25T19:32:18.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.494+0000 7f4064ff9640 10 client.4378.objecter ms_dispatch 0x7f4068000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:18.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.494+0000 7f4064ff9640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:18.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.494+0000 7f4064ff9640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:18.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.494+0000 7f4064ff9640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T19:32:18.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.752+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40600503a0 con 0x7f40681430c0 2024-03-25T19:32:18.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.752+0000 7f4064ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:18.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:18.752+0000 7f4064ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:19.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.008+0000 7f406d8a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:19.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.008+0000 7f406d8a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:19.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.008+0000 7f406d8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:19.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.008+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f406813b430 con 0x7f40681430c0 2024-03-25T19:32:19.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.035+0000 7f4064ff9640 1 -- 172.21.15.134:0/2506801261 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f40600524a0 con 0x7f40681430c0 2024-03-25T19:32:19.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.035+0000 7f4064ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:19.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.035+0000 7f4064ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:19.036 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:19.036 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 10 librados: watch_flush enter 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 10 librados: watch_flush exit 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 10 client.4378.objecter successfully canceled tick 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 >> v1:172.21.15.159:6800/1875874887 conn(0x7f404806def0 legacy=0x7f40480703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 10 monclient: shutdown 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 >> v1:172.21.15.159:6789/0 conn(0x7f40681430c0 legacy=0x7f40681e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 shutdown_connections 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 >> 172.21.15.134:0/2506801261 conn(0x7f40680b7ea0 msgr2=0x7f4068074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 shutdown_connections 2024-03-25T19:32:19.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.038+0000 7f406d8a8640 1 -- 172.21.15.134:0/2506801261 wait complete. 2024-03-25T19:32:19.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:19.039+0000 7f406d8a8640 1 librados: shutdown 2024-03-25T19:32:19.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:25.859 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:25.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.974+0000 7f451719d640 10 monclient: get_monmap_and_config 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.974+0000 7f451719d640 10 monclient: build_initial_monmap 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.975+0000 7f451719d640 10 monclient: monmap: 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:25.976043+0000 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:25.976043+0000 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:25.975 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.975+0000 7f451719d640 1 Processor -- start 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 1 -- start start 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient: init 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient(hunting): picked mon.noname-c con 0x7f451013f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient(hunting): picked mon.noname-a con 0x7f45101430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510142520 con 0x7f45101430c0 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b28a0 con 0x7f451013f710 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100b2a30 con 0x7f45100a98f0 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:25.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.976+0000 7f451719d640 10 monclient(hunting): authenticate will time out at 6178.135448s 2024-03-25T19:32:25.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.991+0000 7f4514f39640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f45100a98f0 0x7f45100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33168/0 (socket says 172.21.15.134:33168) 2024-03-25T19:32:25.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:25.991+0000 7f4514f39640 1 -- 172.21.15.134:0/2746607509 learned_addr learned my addr 172.21.15.134:0/2746607509 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:26.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.001+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355983845 0 0) 0x7f45100b2a30 con 0x7f45100a98f0 2024-03-25T19:32:26.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.001+0000 7f450f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:26.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.001+0000 7f450f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:26.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.001+0000 7f450f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:26.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.001+0000 7f450f7fe640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:32:26.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.001+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4500003520 con 0x7f45100a98f0 2024-03-25T19:32:26.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.008+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498301507 0 0) 0x7f4510142520 con 0x7f45101430c0 2024-03-25T19:32:26.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.008+0000 7f450f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:26.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.008+0000 7f450f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:26.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.008+0000 7f450f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:26.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.008+0000 7f450f7fe640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:32:26.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.008+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45100b2a30 con 0x7f45101430c0 2024-03-25T19:32:26.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.014+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935395854 0 0) 0x7f45100b28a0 con 0x7f451013f710 2024-03-25T19:32:26.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.014+0000 7f450f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:26.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.014+0000 7f450f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:26.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.014+0000 7f450f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:26.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.014+0000 7f450f7fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:32:26.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.014+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4510142520 con 0x7f451013f710 2024-03-25T19:32:26.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3613030733 0 0) 0x7f4500003520 con 0x7f45100a98f0 2024-03-25T19:32:26.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 >> v1:172.21.15.159:6789/0 conn(0x7f451013f710 legacy=0x7f45101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:26.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 >> v1:172.21.15.134:6789/0 conn(0x7f45101430c0 legacy=0x7f451014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:26.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:32:26.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 1 monclient: found mon.noname-b 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100b44c0 con 0x7f45100a98f0 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4504001690 con 0x7f45100a98f0 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f451719d640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f451719d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f450f7fe640 10 monclient: dump: 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.016 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:26.017 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:26.017 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:26.017 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:26.017 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:26.017 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:26.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.015+0000 7f451719d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:26.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45040017f0 con 0x7f45100a98f0 2024-03-25T19:32:26.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f450f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:26.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f450f7fe640 1 -- 172.21.15.134:0/2746607509 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4504001c00 con 0x7f45100a98f0 2024-03-25T19:32:26.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f451719d640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:26.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f451719d640 4 set_mon_vals no callback set 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f450f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f450f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.018+0000 7f450f7fe640 10 monclient: dump: 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:26.019 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.019+0000 7f451719d640 10 monclient: shutdown 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.019+0000 7f451719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.019+0000 7f451719d640 1 -- 172.21.15.134:0/2746607509 >> v1:172.21.15.134:6790/0 conn(0x7f45100a98f0 legacy=0x7f45100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.019+0000 7f451719d640 1 -- 172.21.15.134:0/2746607509 shutdown_connections 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.019+0000 7f451719d640 1 -- 172.21.15.134:0/2746607509 >> 172.21.15.134:0/2746607509 conn(0x7f45100b7ea0 msgr2=0x7f45100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.019+0000 7f451719d640 1 -- 172.21.15.134:0/2746607509 shutdown_connections 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.019+0000 7f451719d640 1 -- 172.21.15.134:0/2746607509 wait complete. 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.020+0000 7f451719d640 10 monclient: build_initial_monmap 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.020+0000 7f451719d640 10 monclient: monmap: 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:26.021 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.020+0000 7f451719d640 1 librados: starting msgr at 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.020+0000 7f451719d640 1 librados: starting objecter 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 1 Processor -- start 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 1 -- start start 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 1 librados: setting wanted keys 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 1 librados: calling monclient init 2024-03-25T19:32:26.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 10 monclient: init 2024-03-25T19:32:26.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:26.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:26.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 10 monclient(hunting): picked mon.noname-b con 0x7f45100a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:26.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 10 monclient(hunting): picked mon.noname-c con 0x7f451013f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:26.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 10 monclient(hunting): picked mon.noname-a con 0x7f45101430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:26.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510138b20 con 0x7f45101430c0 2024-03-25T19:32:26.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.021+0000 7f451719d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510137a80 con 0x7f45100a98f0 2024-03-25T19:32:26.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.022+0000 7f451719d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510139880 con 0x7f451013f710 2024-03-25T19:32:26.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.022+0000 7f451719d640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:26.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.022+0000 7f451719d640 10 monclient(hunting): authenticate will time out at 6178.180650s 2024-03-25T19:32:26.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.033+0000 7f450ffff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f451013f710 0x7f4510139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33208/0 (socket says 172.21.15.134:33208) 2024-03-25T19:32:26.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.033+0000 7f450ffff640 1 -- 172.21.15.134:0/2679223571 learned_addr learned my addr 172.21.15.134:0/2679223571 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:26.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.035+0000 7f450d7fa640 10 client.?.objecter ms_handle_connect 0x7f451013f710 2024-03-25T19:32:26.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.035+0000 7f450d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:26.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.038+0000 7f450d7fa640 10 client.?.objecter ms_handle_connect 0x7f45100a98f0 2024-03-25T19:32:26.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.038+0000 7f450d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:26.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.047+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389954517 0 0) 0x7f4510137a80 con 0x7f45100a98f0 2024-03-25T19:32:26.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.047+0000 7f450d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:32:26.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.047+0000 7f450d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:26.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.047+0000 7f450d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:26.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.047+0000 7f450d7fa640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:32:26.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.047+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f4004eb0 con 0x7f45100a98f0 2024-03-25T19:32:26.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.052+0000 7f450d7fa640 10 client.?.objecter ms_handle_connect 0x7f45101430c0 2024-03-25T19:32:26.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.052+0000 7f450d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:26.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.055+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156029576 0 0) 0x7f4510139880 con 0x7f451013f710 2024-03-25T19:32:26.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.055+0000 7f450d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:32:26.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.055+0000 7f450d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:26.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.055+0000 7f450d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:26.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.055+0000 7f450d7fa640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:32:26.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.055+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4510137a80 con 0x7f451013f710 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.070+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171897077 0 0) 0x7f44f4004eb0 con 0x7f45100a98f0 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.070+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4510139880 con 0x7f45100a98f0 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.070+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4504001b00 con 0x7f45100a98f0 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.070+0000 7f450d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.070+0000 7f450d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.070+0000 7f450d7fa640 10 monclient(hunting): dump: 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:26.070 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:26.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.070+0000 7f450d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.075+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546319690 0 0) 0x7f4510137a80 con 0x7f451013f710 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.075+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44f4004eb0 con 0x7f451013f710 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.075+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f44fc0019f0 con 0x7f451013f710 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.075+0000 7f450d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.075+0000 7f450d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.075+0000 7f450d7fa640 10 monclient(hunting): dump: 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:26.075 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.076+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847617296 0 0) 0x7f4510138b20 con 0x7f45101430c0 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.076+0000 7f450d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.076+0000 7f450d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.076+0000 7f450d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.076+0000 7f450d7fa640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:32:26.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.076+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4510137a80 con 0x7f45101430c0 2024-03-25T19:32:26.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2843293385 0 0) 0x7f44f4004eb0 con 0x7f451013f710 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 >> v1:172.21.15.159:6789/0 conn(0x7f45100a98f0 legacy=0x7f4510137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 >> v1:172.21.15.134:6789/0 conn(0x7f45101430c0 legacy=0x7f45101e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 1 monclient: found mon.c 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4510139a10 con 0x7f451013f710 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f450d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 5 monclient: authenticate success, global_id 4388 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 10 monclient: _renew_subs 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:26.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45101dab00 con 0x7f451013f710 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 10 monclient: _renew_subs 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f451013b4f0 con 0x7f451013f710 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.105+0000 7f451719d640 1 librados: init done 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.106+0000 7f451719d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.106+0000 7f451719d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:26.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.106+0000 7f451719d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:26.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.106+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45101dae00 con 0x7f451013f710 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44fc003610 con 0x7f451013f710 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f44fc0039a0 con 0x7f451013f710 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450d7fa640 10 monclient: dump: 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:26.440 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:26.441 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:26.441 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:26.441 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:26.441 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:26.441 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:26.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450effd640 4 set_mon_vals no callback set 2024-03-25T19:32:26.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.440+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f44fc003bc0 con 0x7f451013f710 2024-03-25T19:32:26.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.441+0000 7f4514f39640 10 monclient: get_auth_request con 0x7f44f406dee0 auth_method 0 2024-03-25T19:32:26.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.441+0000 7f450d7fa640 10 client.4388.objecter ms_handle_connect 0x7f44f406dee0 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f44fc054840 con 0x7f451013f710 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 10 client.4388.objecter ms_dispatch 0x7f4510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44fc050350 con 0x7f451013f710 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:26.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.446+0000 7f450d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:26.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.654+0000 7f451719d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:26.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.654+0000 7f451719d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:26.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.654+0000 7f451719d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:26.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.654+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45100b3e80 con 0x7f451013f710 2024-03-25T19:32:26.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.751+0000 7f450d7fa640 1 -- 172.21.15.134:0/2679223571 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f44fc052280 con 0x7f451013f710 2024-03-25T19:32:26.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.751+0000 7f450d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:26.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.751+0000 7f450d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:26.751 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:26.751 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 10 librados: watch_flush enter 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 10 librados: watch_flush exit 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 10 client.4388.objecter successfully canceled tick 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 >> v1:172.21.15.159:6800/1875874887 conn(0x7f44f406dee0 legacy=0x7f44f40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 10 monclient: shutdown 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.753+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 >> v1:172.21.15.134:6790/0 conn(0x7f451013f710 legacy=0x7f4510139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.754+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 shutdown_connections 2024-03-25T19:32:26.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.754+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 >> 172.21.15.134:0/2679223571 conn(0x7f45100b7ea0 msgr2=0x7f4510074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:26.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.754+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 shutdown_connections 2024-03-25T19:32:26.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.754+0000 7f451719d640 1 -- 172.21.15.134:0/2679223571 wait complete. 2024-03-25T19:32:26.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:26.754+0000 7f451719d640 1 librados: shutdown 2024-03-25T19:32:26.766 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:33.572 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.688+0000 7f6ed0d15640 10 monclient: get_monmap_and_config 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.688+0000 7f6ed0d15640 10 monclient: build_initial_monmap 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.688+0000 7f6ed0d15640 10 monclient: monmap: 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:33.689694+0000 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:33.689694+0000 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:33.688 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:33.689 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:33.689 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:33.689 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:33.689 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.689+0000 7f6ed0d15640 1 Processor -- start 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.689+0000 7f6ed0d15640 1 -- start start 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.689+0000 7f6ed0d15640 10 monclient: init 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 10 monclient(hunting): picked mon.noname-a con 0x7f6ecc0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 10 monclient(hunting): picked mon.noname-c con 0x7f6ecc0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 10 monclient(hunting): picked mon.noname-b con 0x7f6ecc14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc0af940 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc0b0a70 con 0x7f6ecc0b1730 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc0b0d50 con 0x7f6ecc14be90 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:33.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.690+0000 7f6ed0d15640 10 monclient(hunting): authenticate will time out at 6185.848923s 2024-03-25T19:32:33.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.698+0000 7f6ecad9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f6ecc14be90 0x7f6ecc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:42750/0 (socket says 172.21.15.134:42750) 2024-03-25T19:32:33.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.698+0000 7f6ecad9d640 1 -- 172.21.15.134:0/3349869571 learned_addr learned my addr 172.21.15.134:0/3349869571 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:33.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.708+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627472348 0 0) 0x7f6ecc0b0a70 con 0x7f6ecc0b1730 2024-03-25T19:32:33.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.708+0000 7f6ec959a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:33.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.708+0000 7f6ec959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:33.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.708+0000 7f6ec959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:33.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.708+0000 7f6ec959a640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:32:33.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.708+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb4003520 con 0x7f6ecc0b1730 2024-03-25T19:32:33.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.716+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353321252 0 0) 0x7f6ecc0af940 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.716+0000 7f6ec959a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:33.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.716+0000 7f6ec959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:33.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.716+0000 7f6ec959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:33.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.716+0000 7f6ec959a640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:32:33.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.716+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ecc0b0a70 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202073104 0 0) 0x7f6eb4003520 con 0x7f6ecc0b1730 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 >> v1:172.21.15.134:6790/0 conn(0x7f6ecc14be90 legacy=0x7f6ecc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 >> v1:172.21.15.134:6789/0 conn(0x7f6ecc0b4d60 legacy=0x7f6ecc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 10 monclient: _finish_hunting 0 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 1 monclient: found mon.noname-c 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc0b1030 con 0x7f6ecc0b1730 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 10 monclient: _finish_auth 0 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 10 monclient: _check_auth_tickets 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6ec0001690 con 0x7f6ecc0b1730 2024-03-25T19:32:33.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ec959a640 10 monclient: dump: 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ed0d15640 5 monclient: authenticate success, global_id 4387 2024-03-25T19:32:33.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.720+0000 7f6ed0d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:33.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ec00017f0 con 0x7f6ecc0b1730 2024-03-25T19:32:33.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ec959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:33.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ec959a640 1 -- 172.21.15.134:0/3349869571 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6ec0001c00 con 0x7f6ecc0b1730 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ed0d15640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ed0d15640 4 set_mon_vals no callback set 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ec959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ec959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.734+0000 7f6ec959a640 10 monclient: dump: 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:33.735 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.735+0000 7f6ed0d15640 10 monclient: shutdown 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.735+0000 7f6ed0d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.735+0000 7f6ed0d15640 1 -- 172.21.15.134:0/3349869571 >> v1:172.21.15.159:6789/0 conn(0x7f6ecc0b1730 legacy=0x7f6ecc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.735+0000 7f6ed0d15640 1 -- 172.21.15.134:0/3349869571 shutdown_connections 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.735+0000 7f6ed0d15640 1 -- 172.21.15.134:0/3349869571 >> 172.21.15.134:0/3349869571 conn(0x7f6ecc0617a0 msgr2=0x7f6ecc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.735+0000 7f6ed0d15640 1 -- 172.21.15.134:0/3349869571 shutdown_connections 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.735+0000 7f6ed0d15640 1 -- 172.21.15.134:0/3349869571 wait complete. 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.736+0000 7f6ed0d15640 10 monclient: build_initial_monmap 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.736+0000 7f6ed0d15640 10 monclient: monmap: 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:33.737 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 1 librados: starting msgr at 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 1 librados: starting objecter 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 1 Processor -- start 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 1 -- start start 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 1 librados: setting wanted keys 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 1 librados: calling monclient init 2024-03-25T19:32:33.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 10 monclient: init 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 10 monclient(hunting): picked mon.noname-a con 0x7f6ecc0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.737+0000 7f6ed0d15640 10 monclient(hunting): picked mon.noname-b con 0x7f6ecc0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.738+0000 7f6ed0d15640 10 monclient(hunting): picked mon.noname-c con 0x7f6ecc14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.738+0000 7f6ed0d15640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc141790 con 0x7f6ecc0b1730 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.738+0000 7f6ed0d15640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc141a00 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.738+0000 7f6ed0d15640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc1d7a90 con 0x7f6ecc14be90 2024-03-25T19:32:33.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.738+0000 7f6ed0d15640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:33.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.738+0000 7f6ed0d15640 10 monclient(hunting): authenticate will time out at 6185.896737s 2024-03-25T19:32:33.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.751+0000 7f6ec9d9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f6ecc0b4d60 0x7f6ecc142a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52248/0 (socket says 172.21.15.134:52248) 2024-03-25T19:32:33.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.751+0000 7f6ec9d9b640 1 -- 172.21.15.134:0/4141031856 learned_addr learned my addr 172.21.15.134:0/4141031856 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:33.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.755+0000 7f6ebaffd640 10 client.?.objecter ms_handle_connect 0x7f6ecc0b4d60 2024-03-25T19:32:33.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.755+0000 7f6ebaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:33.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.758+0000 7f6ebaffd640 10 client.?.objecter ms_handle_connect 0x7f6ecc0b1730 2024-03-25T19:32:33.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.758+0000 7f6ebaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:33.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.762+0000 7f6ebaffd640 10 client.?.objecter ms_handle_connect 0x7f6ecc14be90 2024-03-25T19:32:33.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.762+0000 7f6ebaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:33.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.767+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743498146 0 0) 0x7f6ecc141790 con 0x7f6ecc0b1730 2024-03-25T19:32:33.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.767+0000 7f6ebaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:33.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.767+0000 7f6ebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:33.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.767+0000 7f6ebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:33.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.767+0000 7f6ebaffd640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:32:33.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.767+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eac004eb0 con 0x7f6ecc0b1730 2024-03-25T19:32:33.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.771+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523969592 0 0) 0x7f6ecc141a00 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.771+0000 7f6ebaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:33.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.771+0000 7f6ebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:33.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.771+0000 7f6ebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:33.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.771+0000 7f6ebaffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:32:33.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.771+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ecc141790 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.775+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353767817 0 0) 0x7f6ecc1d7a90 con 0x7f6ecc14be90 2024-03-25T19:32:33.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.775+0000 7f6ebaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:33.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.775+0000 7f6ebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:33.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.775+0000 7f6ebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:33.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.775+0000 7f6ebaffd640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:32:33.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.775+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ecc141a00 con 0x7f6ecc14be90 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.784+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3861578257 0 0) 0x7f6ecc141790 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.784+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ecc1d7a90 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.784+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6ec0001690 con 0x7f6ecc0b4d60 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.784+0000 7f6ebaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.784+0000 7f6ebaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.784+0000 7f6ebaffd640 10 monclient(hunting): dump: 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:33.784 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:33.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.784+0000 7f6ebaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:33.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.786+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3054429021 0 0) 0x7f6eac004eb0 con 0x7f6ecc0b1730 2024-03-25T19:32:33.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.786+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ecc141790 con 0x7f6ecc0b1730 2024-03-25T19:32:33.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.786+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6ebc001930 con 0x7f6ecc0b1730 2024-03-25T19:32:33.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.786+0000 7f6ebaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.786+0000 7f6ebaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.786+0000 7f6ebaffd640 10 monclient(hunting): dump: 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:33.787 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:33.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2098502668 0 0) 0x7f6ecc141790 con 0x7f6ecc0b1730 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 >> v1:172.21.15.134:6790/0 conn(0x7f6ecc14be90 legacy=0x7f6ecc1d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 >> v1:172.21.15.159:6789/0 conn(0x7f6ecc0b4d60 legacy=0x7f6ecc142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 1 monclient: found mon.a 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc143140 con 0x7f6ecc0b1730 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ebaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 5 monclient: authenticate success, global_id 4431 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 10 monclient: _renew_subs 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc1d86c0 con 0x7f6ecc0b1730 2024-03-25T19:32:33.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 10 monclient: _renew_subs 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc1422e0 con 0x7f6ecc0b1730 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.808+0000 7f6ed0d15640 1 librados: init done 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.809+0000 7f6ed0d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.809+0000 7f6ed0d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.809+0000 7f6ed0d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:33.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:33.809+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc1d7c20 con 0x7f6ecc0b1730 2024-03-25T19:32:34.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.108+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ebc003610 con 0x7f6ecc0b1730 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.109+0000 7f6ebaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.109+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6ebc003920 con 0x7f6ecc0b1730 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.109+0000 7f6ebaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.109+0000 7f6ebaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.109+0000 7f6ebaffd640 10 monclient: dump: 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:34.110 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:34.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.109+0000 7f6ec8d99640 4 set_mon_vals no callback set 2024-03-25T19:32:34.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.109+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f6ebc019970 con 0x7f6ecc0b1730 2024-03-25T19:32:34.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.110+0000 7f6ec9d9b640 10 monclient: get_auth_request con 0x7f6eac06dfb0 auth_method 0 2024-03-25T19:32:34.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.110+0000 7f6ebaffd640 10 client.4431.objecter ms_handle_connect 0x7f6eac06dfb0 2024-03-25T19:32:34.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f6ebc053650 con 0x7f6ecc0b1730 2024-03-25T19:32:34.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 10 client.4431.objecter ms_dispatch 0x7f6ecc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:34.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:34.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:34.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-25T19:32:34.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ebc04ef90 con 0x7f6ecc0b1730 2024-03-25T19:32:34.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:34.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.115+0000 7f6ebaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:34.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.372+0000 7f6ed0d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:34.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.372+0000 7f6ed0d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:34.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.372+0000 7f6ed0d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:32:34.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.372+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ecc0627b0 con 0x7f6ecc0b1730 2024-03-25T19:32:34.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.412+0000 7f6ebaffd640 1 -- 172.21.15.134:0/4141031856 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f6ebc051090 con 0x7f6ecc0b1730 2024-03-25T19:32:34.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.412+0000 7f6ebaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:34.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.412+0000 7f6ebaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:34.413 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:34.413 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:34.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.414+0000 7f6ed0d15640 10 librados: watch_flush enter 2024-03-25T19:32:34.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 10 librados: watch_flush exit 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 10 client.4431.objecter successfully canceled tick 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 >> v1:172.21.15.159:6800/1875874887 conn(0x7f6eac06dfb0 legacy=0x7f6eac070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 10 monclient: shutdown 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 >> v1:172.21.15.134:6789/0 conn(0x7f6ecc0b1730 legacy=0x7f6ecc13b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 shutdown_connections 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 >> 172.21.15.134:0/4141031856 conn(0x7f6ecc0617a0 msgr2=0x7f6ecc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 shutdown_connections 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 1 -- 172.21.15.134:0/4141031856 wait complete. 2024-03-25T19:32:34.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:34.415+0000 7f6ed0d15640 1 librados: shutdown 2024-03-25T19:32:34.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:40.221 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:44.893 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.228 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:44.900 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.274 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:44.921 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.277 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:44.949 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.285 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:44.932 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.304 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:44.976 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.313 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:44.985 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.314 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:44.961 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:40.342 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:45.014 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:41.234 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:41.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.348+0000 7fbe39954640 10 monclient: get_monmap_and_config 2024-03-25T19:32:41.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.348+0000 7fbe39954640 10 monclient: build_initial_monmap 2024-03-25T19:32:41.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.348+0000 7fbe39954640 10 monclient: monmap: 2024-03-25T19:32:41.348 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:41.348 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:41.348 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:41.349103+0000 2024-03-25T19:32:41.348 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:41.349103+0000 2024-03-25T19:32:41.349 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:41.349 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:41.349 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:41.349 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:41.349 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:41.349 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.348+0000 7fbe39954640 1 Processor -- start 2024-03-25T19:32:41.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.348+0000 7fbe39954640 1 -- start start 2024-03-25T19:32:41.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.348+0000 7fbe39954640 10 monclient: init 2024-03-25T19:32:41.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 10 monclient(hunting): picked mon.noname-a con 0x7fbe340a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 10 monclient(hunting): picked mon.noname-c con 0x7fbe3413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 10 monclient(hunting): picked mon.noname-b con 0x7fbe341430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe34142520 con 0x7fbe340a98f0 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe340b28a0 con 0x7fbe3413f710 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe340b2a30 con 0x7fbe341430c0 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:41.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.349+0000 7fbe39954640 10 monclient(hunting): authenticate will time out at 6193.508118s 2024-03-25T19:32:41.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.365+0000 7fbe327fc640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbe3413f710 0x7fbe341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:35326/0 (socket says 172.21.15.134:35326) 2024-03-25T19:32:41.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.365+0000 7fbe327fc640 1 -- 172.21.15.134:0/471959468 learned_addr learned my addr 172.21.15.134:0/471959468 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:41.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.373+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087380192 0 0) 0x7fbe340b28a0 con 0x7fbe3413f710 2024-03-25T19:32:41.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.373+0000 7fbe31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:41.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.373+0000 7fbe31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:41.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.373+0000 7fbe31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:41.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.373+0000 7fbe31ffb640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:32:41.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.373+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c003520 con 0x7fbe3413f710 2024-03-25T19:32:41.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.374+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739471773 0 0) 0x7fbe34142520 con 0x7fbe340a98f0 2024-03-25T19:32:41.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.374+0000 7fbe31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:41.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.374+0000 7fbe31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:41.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.374+0000 7fbe31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:41.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.374+0000 7fbe31ffb640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:32:41.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.374+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe340b28a0 con 0x7fbe340a98f0 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1395209108 0 0) 0x7fbe1c003520 con 0x7fbe3413f710 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 >> v1:172.21.15.134:6790/0 conn(0x7fbe341430c0 legacy=0x7fbe3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 >> v1:172.21.15.134:6789/0 conn(0x7fbe340a98f0 legacy=0x7fbe340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 1 monclient: found mon.noname-c 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe340b44c0 con 0x7fbe3413f710 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:41.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe28001690 con 0x7fbe3413f710 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe39954640 5 monclient: authenticate success, global_id 4393 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe39954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe31ffb640 10 monclient: dump: 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.380+0000 7fbe39954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe280017f0 con 0x7fbe3413f710 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe31ffb640 1 -- 172.21.15.134:0/471959468 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe28001c00 con 0x7fbe3413f710 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe39954640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe39954640 4 set_mon_vals no callback set 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.384+0000 7fbe31ffb640 10 monclient: dump: 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.384 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.385 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:41.385 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:41.385 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:41.385 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:41.385 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:41.385 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 10 monclient: shutdown 2024-03-25T19:32:41.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:41.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 1 -- 172.21.15.134:0/471959468 >> v1:172.21.15.159:6789/0 conn(0x7fbe3413f710 legacy=0x7fbe341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 1 -- 172.21.15.134:0/471959468 shutdown_connections 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 1 -- 172.21.15.134:0/471959468 >> 172.21.15.134:0/471959468 conn(0x7fbe340b7ea0 msgr2=0x7fbe340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 1 -- 172.21.15.134:0/471959468 shutdown_connections 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 1 -- 172.21.15.134:0/471959468 wait complete. 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 10 monclient: build_initial_monmap 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.385+0000 7fbe39954640 10 monclient: monmap: 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:41.387 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.386+0000 7fbe39954640 1 librados: starting msgr at 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.386+0000 7fbe39954640 1 librados: starting objecter 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.386+0000 7fbe39954640 1 Processor -- start 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.386+0000 7fbe39954640 1 -- start start 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.386+0000 7fbe39954640 1 librados: setting wanted keys 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.386+0000 7fbe39954640 1 librados: calling monclient init 2024-03-25T19:32:41.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.386+0000 7fbe39954640 10 monclient: init 2024-03-25T19:32:41.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:41.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:41.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 10 monclient(hunting): picked mon.noname-a con 0x7fbe340a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:41.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 10 monclient(hunting): picked mon.noname-b con 0x7fbe3413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:41.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 10 monclient(hunting): picked mon.noname-c con 0x7fbe341430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:41.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe34138a50 con 0x7fbe340a98f0 2024-03-25T19:32:41.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe340ab9e0 con 0x7fbe3413f710 2024-03-25T19:32:41.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe341d7880 con 0x7fbe341430c0 2024-03-25T19:32:41.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:41.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.387+0000 7fbe39954640 10 monclient(hunting): authenticate will time out at 6193.546369s 2024-03-25T19:32:41.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.392+0000 7fbe337fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fbe341430c0 0x7fbe341e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:55312/0 (socket says 172.21.15.134:55312) 2024-03-25T19:32:41.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.392+0000 7fbe337fe640 1 -- 172.21.15.134:0/2797445275 learned_addr learned my addr 172.21.15.134:0/2797445275 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:41.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.397+0000 7fbe1bfff640 10 client.?.objecter ms_handle_connect 0x7fbe3413f710 2024-03-25T19:32:41.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.397+0000 7fbe1bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:41.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.403+0000 7fbe1bfff640 10 client.?.objecter ms_handle_connect 0x7fbe340a98f0 2024-03-25T19:32:41.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.403+0000 7fbe1bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:41.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.404+0000 7fbe1bfff640 10 client.?.objecter ms_handle_connect 0x7fbe341430c0 2024-03-25T19:32:41.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.404+0000 7fbe1bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:41.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.409+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720383943 0 0) 0x7fbe34138a50 con 0x7fbe340a98f0 2024-03-25T19:32:41.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.409+0000 7fbe1bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:41.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.409+0000 7fbe1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:41.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.409+0000 7fbe1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:41.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.409+0000 7fbe1bfff640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:32:41.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.409+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe20004eb0 con 0x7fbe340a98f0 2024-03-25T19:32:41.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.412+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407381699 0 0) 0x7fbe340ab9e0 con 0x7fbe3413f710 2024-03-25T19:32:41.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.412+0000 7fbe1bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:41.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.412+0000 7fbe1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:41.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.412+0000 7fbe1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:41.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.412+0000 7fbe1bfff640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:32:41.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.412+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe34138a50 con 0x7fbe3413f710 2024-03-25T19:32:41.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.416+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149941636 0 0) 0x7fbe341d7880 con 0x7fbe341430c0 2024-03-25T19:32:41.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.416+0000 7fbe1bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:41.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.416+0000 7fbe1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:41.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.416+0000 7fbe1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:41.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.416+0000 7fbe1bfff640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:32:41.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.416+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe340ab9e0 con 0x7fbe341430c0 2024-03-25T19:32:41.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.426+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3758358554 0 0) 0x7fbe34138a50 con 0x7fbe3413f710 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.427+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe341d7880 con 0x7fbe3413f710 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.427+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe28001b00 con 0x7fbe3413f710 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.427+0000 7fbe1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.427+0000 7fbe1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.427+0000 7fbe1bfff640 10 monclient(hunting): dump: 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.427+0000 7fbe1bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:41.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.431+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1610238357 0 0) 0x7fbe20004eb0 con 0x7fbe340a98f0 2024-03-25T19:32:41.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.431+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe34138a50 con 0x7fbe340a98f0 2024-03-25T19:32:41.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.431+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe24001930 con 0x7fbe340a98f0 2024-03-25T19:32:41.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.431+0000 7fbe1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:41.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.431+0000 7fbe1bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.431+0000 7fbe1bfff640 10 monclient(hunting): dump: 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:41.432 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278966815 0 0) 0x7fbe340ab9e0 con 0x7fbe341430c0 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe20004eb0 con 0x7fbe341430c0 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe2c001bd0 con 0x7fbe341430c0 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 10 monclient(hunting): dump: 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:41.436 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 843358574 0 0) 0x7fbe341d7880 con 0x7fbe3413f710 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 >> v1:172.21.15.134:6790/0 conn(0x7fbe341430c0 legacy=0x7fbe341e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 >> v1:172.21.15.134:6789/0 conn(0x7fbe340a98f0 legacy=0x7fbe34138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 monclient: found mon.b 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe341d84b0 con 0x7fbe3413f710 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 10 monclient: _finish_auth 0 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 10 monclient: _renew_subs 2024-03-25T19:32:41.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:41.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe341d51f0 con 0x7fbe3413f710 2024-03-25T19:32:41.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:41.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 10 monclient: _renew_subs 2024-03-25T19:32:41.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:41.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe34138e60 con 0x7fbe3413f710 2024-03-25T19:32:41.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.435+0000 7fbe39954640 1 librados: init done 2024-03-25T19:32:41.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.436+0000 7fbe39954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:41.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.436+0000 7fbe39954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:41.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.436+0000 7fbe39954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:41.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.436+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe341d76e0 con 0x7fbe3413f710 2024-03-25T19:32:41.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.728+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe28004a90 con 0x7fbe3413f710 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.728+0000 7fbe1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.728+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fbe28004c00 con 0x7fbe3413f710 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.728+0000 7fbe1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.728+0000 7fbe1bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.728+0000 7fbe1bfff640 10 monclient: dump: 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:41.729 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:41.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.728+0000 7fbe317fa640 4 set_mon_vals no callback set 2024-03-25T19:32:41.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.729+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fbe2801abd0 con 0x7fbe3413f710 2024-03-25T19:32:41.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.730+0000 7fbe32ffd640 10 monclient: get_auth_request con 0x7fbe2006e0b0 auth_method 0 2024-03-25T19:32:41.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.730+0000 7fbe1bfff640 10 client.4396.objecter ms_handle_connect 0x7fbe2006e0b0 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fbe28054640 con 0x7fbe3413f710 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 10 client.4396.objecter ms_dispatch 0x7fbe34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe28050150 con 0x7fbe3413f710 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:41.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.735+0000 7fbe1bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:41.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.981+0000 7fbe39954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:41.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.981+0000 7fbe39954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:41.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.981+0000 7fbe39954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:41.981+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe340627b0 con 0x7fbe3413f710 2024-03-25T19:32:42.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.010+0000 7fbe1bfff640 1 -- 172.21.15.134:0/2797445275 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fbe28052080 con 0x7fbe3413f710 2024-03-25T19:32:42.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.011+0000 7fbe1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:42.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.011+0000 7fbe1bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:42.011 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:42.011 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 10 librados: watch_flush enter 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 10 librados: watch_flush exit 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 >> v1:172.21.15.159:6800/1875874887 conn(0x7fbe2006e0b0 legacy=0x7fbe20070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 10 monclient: shutdown 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:42.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 >> v1:172.21.15.159:6789/0 conn(0x7fbe3413f710 legacy=0x7fbe341d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:42.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.013+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 shutdown_connections 2024-03-25T19:32:42.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.014+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 >> 172.21.15.134:0/2797445275 conn(0x7fbe340b7ea0 msgr2=0x7fbe34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:42.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.014+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 shutdown_connections 2024-03-25T19:32:42.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.014+0000 7fbe39954640 1 -- 172.21.15.134:0/2797445275 wait complete. 2024-03-25T19:32:42.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:42.014+0000 7fbe39954640 1 librados: shutdown 2024-03-25T19:32:42.027 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:48.831 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:48.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.951+0000 7f3c296ae640 10 monclient: get_monmap_and_config 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.951+0000 7f3c296ae640 10 monclient: build_initial_monmap 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.952+0000 7f3c296ae640 10 monclient: monmap: 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:48.953058+0000 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:48.953058+0000 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:48.952 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:48.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.952+0000 7f3c296ae640 1 Processor -- start 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.952+0000 7f3c296ae640 1 -- start start 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.952+0000 7f3c296ae640 10 monclient: init 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3c24061830 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3c2413db20 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3c240b0510 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2406eba0 con 0x7f3c240b0510 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c240b0960 con 0x7f3c24061830 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c24072830 con 0x7f3c2413db20 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:48.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.953+0000 7f3c296ae640 10 monclient(hunting): authenticate will time out at 6201.112220s 2024-03-25T19:32:48.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.960+0000 7f3c22ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f3c24061830 0x7f3c2413d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:40124/0 (socket says 172.21.15.134:40124) 2024-03-25T19:32:48.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.961+0000 7f3c22ffd640 1 -- 172.21.15.134:0/147720982 learned_addr learned my addr 172.21.15.134:0/147720982 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:48.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.980+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179259165 0 0) 0x7f3c24072830 con 0x7f3c2413db20 2024-03-25T19:32:48.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.980+0000 7f3c21ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:48.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.980+0000 7f3c21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:48.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.980+0000 7f3c21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:48.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.980+0000 7f3c21ffb640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:32:48.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.980+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c0c003520 con 0x7f3c2413db20 2024-03-25T19:32:48.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.983+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716908088 0 0) 0x7f3c2406eba0 con 0x7f3c240b0510 2024-03-25T19:32:48.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.983+0000 7f3c21ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:48.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.983+0000 7f3c21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:48.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.983+0000 7f3c21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:48.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.983+0000 7f3c21ffb640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:32:48.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.983+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c24072830 con 0x7f3c240b0510 2024-03-25T19:32:48.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.987+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253474746 0 0) 0x7f3c240b0960 con 0x7f3c24061830 2024-03-25T19:32:48.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.987+0000 7f3c21ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:48.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.987+0000 7f3c21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:48.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.987+0000 7f3c21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:48.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.987+0000 7f3c21ffb640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:32:48.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.987+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c2406eba0 con 0x7f3c24061830 2024-03-25T19:32:48.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786628934 0 0) 0x7f3c24072830 con 0x7f3c240b0510 2024-03-25T19:32:48.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 >> v1:172.21.15.134:6790/0 conn(0x7f3c2413db20 legacy=0x7f3c240afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:48.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 >> v1:172.21.15.159:6789/0 conn(0x7f3c24061830 legacy=0x7f3c2413d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:48.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:48.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 1 monclient: found mon.noname-a 2024-03-25T19:32:48.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:48.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c24072b10 con 0x7f3c240b0510 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3c1c0019f0 con 0x7f3c240b0510 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c296ae640 5 monclient: authenticate success, global_id 4440 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c296ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c21ffb640 10 monclient: dump: 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:48.995 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:48.996 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:48.996 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:48.996 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:48.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:48.994+0000 7f3c296ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c1c001b50 con 0x7f3c240b0510 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c21ffb640 1 -- 172.21.15.134:0/147720982 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3c1c001ee0 con 0x7f3c240b0510 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c21ffb640 10 monclient: dump: 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:49.007 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c296ae640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:49.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.007+0000 7f3c296ae640 4 set_mon_vals no callback set 2024-03-25T19:32:49.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.008+0000 7f3c296ae640 10 monclient: shutdown 2024-03-25T19:32:49.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.008+0000 7f3c296ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.008+0000 7f3c296ae640 1 -- 172.21.15.134:0/147720982 >> v1:172.21.15.134:6789/0 conn(0x7f3c240b0510 legacy=0x7f3c2414ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.008+0000 7f3c296ae640 1 -- 172.21.15.134:0/147720982 shutdown_connections 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.008+0000 7f3c296ae640 1 -- 172.21.15.134:0/147720982 >> 172.21.15.134:0/147720982 conn(0x7f3c240b0ef0 msgr2=0x7f3c240b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.008+0000 7f3c296ae640 1 -- 172.21.15.134:0/147720982 shutdown_connections 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.008+0000 7f3c296ae640 1 -- 172.21.15.134:0/147720982 wait complete. 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.009+0000 7f3c296ae640 10 monclient: build_initial_monmap 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.009+0000 7f3c296ae640 10 monclient: monmap: 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:49.010 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.009+0000 7f3c296ae640 1 librados: starting msgr at 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.009+0000 7f3c296ae640 1 librados: starting objecter 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.010+0000 7f3c296ae640 1 Processor -- start 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.010+0000 7f3c296ae640 1 -- start start 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.010+0000 7f3c296ae640 1 librados: setting wanted keys 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.010+0000 7f3c296ae640 1 librados: calling monclient init 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.010+0000 7f3c296ae640 10 monclient: init 2024-03-25T19:32:49.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.010+0000 7f3c296ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3c24061830 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3c240b0510 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3c2413db20 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2413b7a0 con 0x7f3c2413db20 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2414c600 con 0x7f3c240b0510 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c240ad890 con 0x7f3c24061830 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:49.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.011+0000 7f3c296ae640 10 monclient(hunting): authenticate will time out at 6201.169714s 2024-03-25T19:32:49.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.015+0000 7f3c237fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3c2413db20 0x7f3c241e6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:37332/0 (socket says 172.21.15.134:37332) 2024-03-25T19:32:49.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.015+0000 7f3c237fe640 1 -- 172.21.15.134:0/2210214766 learned_addr learned my addr 172.21.15.134:0/2210214766 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:49.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.017+0000 7f3c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c2413db20 2024-03-25T19:32:49.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.017+0000 7f3c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:49.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.017+0000 7f3c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c240b0510 2024-03-25T19:32:49.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.017+0000 7f3c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.027+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598322807 0 0) 0x7f3c2413b7a0 con 0x7f3c2413db20 2024-03-25T19:32:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.027+0000 7f3c0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.027+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.027+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.027+0000 7f3c0b7fe640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:32:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.027+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c10004eb0 con 0x7f3c2413db20 2024-03-25T19:32:49.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.029+0000 7f3c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c24061830 2024-03-25T19:32:49.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.029+0000 7f3c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:49.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.034+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653662406 0 0) 0x7f3c2414c600 con 0x7f3c240b0510 2024-03-25T19:32:49.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.034+0000 7f3c0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:49.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.034+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:49.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.034+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:49.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.034+0000 7f3c0b7fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:32:49.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.034+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c2413b7a0 con 0x7f3c240b0510 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.045+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 996059113 0 0) 0x7f3c2413b7a0 con 0x7f3c240b0510 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.045+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c2414c600 con 0x7f3c240b0510 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.045+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3c18001770 con 0x7f3c240b0510 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.045+0000 7f3c0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.045+0000 7f3c0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.045+0000 7f3c0b7fe640 10 monclient(hunting): dump: 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:49.046 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:49.046 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:49.046 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:49.046 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:49.046 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:49.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.045+0000 7f3c0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:49.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.048+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069018342 0 0) 0x7f3c240ad890 con 0x7f3c24061830 2024-03-25T19:32:49.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.048+0000 7f3c0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:49.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.048+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:49.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.048+0000 7f3c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:49.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.048+0000 7f3c0b7fe640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:32:49.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.048+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c2413b7a0 con 0x7f3c24061830 2024-03-25T19:32:49.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278270289 0 0) 0x7f3c10004eb0 con 0x7f3c2413db20 2024-03-25T19:32:49.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c240ad890 con 0x7f3c2413db20 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3c1c001e60 con 0x7f3c2413db20 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 10 monclient(hunting): dump: 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:49.058 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1406809216 0 0) 0x7f3c2414c600 con 0x7f3c240b0510 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 >> v1:172.21.15.134:6790/0 conn(0x7f3c24061830 legacy=0x7f3c2414bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 >> v1:172.21.15.134:6789/0 conn(0x7f3c2413db20 legacy=0x7f3c241e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 monclient: found mon.b 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c241df360 con 0x7f3c240b0510 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:32:49.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 10 monclient: _renew_subs 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2413d760 con 0x7f3c240b0510 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 10 monclient: _renew_subs 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c24071660 con 0x7f3c240b0510 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.058+0000 7f3c296ae640 1 librados: init done 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.060+0000 7f3c296ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.060+0000 7f3c296ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.060+0000 7f3c296ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:49.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.060+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2413b5b0 con 0x7f3c240b0510 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.362+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c18001c60 con 0x7f3c240b0510 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.362+0000 7f3c0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.362+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f3c180037b0 con 0x7f3c240b0510 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.362+0000 7f3c0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.362+0000 7f3c0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.362+0000 7f3c0b7fe640 10 monclient: dump: 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:49.362 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.362+0000 7f3c217fa640 4 set_mon_vals no callback set 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.363+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f3c180039d0 con 0x7f3c240b0510 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.363+0000 7f3c22ffd640 10 monclient: get_auth_request con 0x7f3c1006dfd0 auth_method 0 2024-03-25T19:32:49.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.364+0000 7f3c0b7fe640 10 client.4402.objecter ms_handle_connect 0x7f3c1006dfd0 2024-03-25T19:32:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f3c180533e0 con 0x7f3c240b0510 2024-03-25T19:32:49.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 10 client.4402.objecter ms_dispatch 0x7f3c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:49.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:49.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:49.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:32:49.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c1804eef0 con 0x7f3c240b0510 2024-03-25T19:32:49.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:49.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.368+0000 7f3c0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:49.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.624+0000 7f3c296ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:49.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.624+0000 7f3c296ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:49.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.624+0000 7f3c296ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:32:49.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.624+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c2413d140 con 0x7f3c240b0510 2024-03-25T19:32:49.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.650+0000 7f3c0b7fe640 1 -- 172.21.15.134:0/2210214766 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f3c18050e20 con 0x7f3c240b0510 2024-03-25T19:32:49.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.650+0000 7f3c0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:49.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.650+0000 7f3c0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:49.650 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:49.650 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 10 librados: watch_flush enter 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 10 librados: watch_flush exit 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 >> v1:172.21.15.159:6800/1875874887 conn(0x7f3c1006dfd0 legacy=0x7f3c10070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 10 monclient: shutdown 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 >> v1:172.21.15.159:6789/0 conn(0x7f3c240b0510 legacy=0x7f3c241dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 shutdown_connections 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.652+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 >> 172.21.15.134:0/2210214766 conn(0x7f3c240b0ef0 msgr2=0x7f3c240afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.653+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 shutdown_connections 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.653+0000 7f3c296ae640 1 -- 172.21.15.134:0/2210214766 wait complete. 2024-03-25T19:32:49.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:49.653+0000 7f3c296ae640 1 librados: shutdown 2024-03-25T19:32:49.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:56.472 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.577+0000 7ff3a0645640 10 monclient: get_monmap_and_config 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.577+0000 7ff3a0645640 10 monclient: build_initial_monmap 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.577+0000 7ff3a0645640 10 monclient: monmap: 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:32:56.578411+0000 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:32:56.578411+0000 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:56.577 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:56.578 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:56.578 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:32:56.578 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:32:56.578 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.577+0000 7ff3a0645640 1 Processor -- start 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 1 -- start start 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient: init 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient(hunting): picked mon.noname-c con 0x7ff3980b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient(hunting): picked mon.noname-b con 0x7ff3980b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:56.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient(hunting): picked mon.noname-a con 0x7ff39814be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:56.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3980af940 con 0x7ff39814be90 2024-03-25T19:32:56.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3980b0a70 con 0x7ff3980b4d60 2024-03-25T19:32:56.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3980b0d50 con 0x7ff3980b1730 2024-03-25T19:32:56.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:56.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.578+0000 7ff3a0645640 10 monclient(hunting): authenticate will time out at 6208.737437s 2024-03-25T19:32:56.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.614+0000 7ff39e3e1640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff3980b4d60 0x7ff39813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:51412/0 (socket says 172.21.15.134:51412) 2024-03-25T19:32:56.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.614+0000 7ff39e3e1640 1 -- 172.21.15.134:0/1555808279 learned_addr learned my addr 172.21.15.134:0/1555808279 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:56.629 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.630+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252946209 0 0) 0x7ff3980b0a70 con 0x7ff3980b4d60 2024-03-25T19:32:56.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.630+0000 7ff39d3df640 10 monclient(hunting): _negotiate 2024-03-25T19:32:56.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.630+0000 7ff39d3df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:56.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.630+0000 7ff39d3df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:56.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.630+0000 7ff39d3df640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:32:56.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.630+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff384003520 con 0x7ff3980b4d60 2024-03-25T19:32:56.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425875106 0 0) 0x7ff384003520 con 0x7ff3980b4d60 2024-03-25T19:32:56.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 >> v1:172.21.15.134:6790/0 conn(0x7ff3980b1730 legacy=0x7ff3980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:56.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 >> v1:172.21.15.134:6789/0 conn(0x7ff39814be90 legacy=0x7ff39814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:56.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 10 monclient: _finish_hunting 0 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 1 monclient: found mon.noname-c 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3980b1030 con 0x7ff3980b4d60 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 10 monclient: _finish_auth 0 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 10 monclient: _check_auth_tickets 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff394001690 con 0x7ff3980b4d60 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff3a0645640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff3a0645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff39d3df640 10 monclient: dump: 2024-03-25T19:32:56.649 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:56.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.648+0000 7ff3a0645640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:56.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3940017f0 con 0x7ff3980b4d60 2024-03-25T19:32:56.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff39d3df640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:56.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff39d3df640 1 -- 172.21.15.134:0/1555808279 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff394001c00 con 0x7ff3980b4d60 2024-03-25T19:32:56.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff39d3df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:56.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff39d3df640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:56.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff39d3df640 10 monclient: dump: 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff3a0645640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:56.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.655+0000 7ff3a0645640 4 set_mon_vals no callback set 2024-03-25T19:32:56.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.656+0000 7ff3a0645640 10 monclient: shutdown 2024-03-25T19:32:56.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.656+0000 7ff3a0645640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.656+0000 7ff3a0645640 1 -- 172.21.15.134:0/1555808279 >> v1:172.21.15.159:6789/0 conn(0x7ff3980b4d60 legacy=0x7ff39813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.656+0000 7ff3a0645640 1 -- 172.21.15.134:0/1555808279 shutdown_connections 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.656+0000 7ff3a0645640 1 -- 172.21.15.134:0/1555808279 >> 172.21.15.134:0/1555808279 conn(0x7ff3980617a0 msgr2=0x7ff398061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.656+0000 7ff3a0645640 1 -- 172.21.15.134:0/1555808279 shutdown_connections 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.656+0000 7ff3a0645640 1 -- 172.21.15.134:0/1555808279 wait complete. 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.657+0000 7ff3a0645640 10 monclient: build_initial_monmap 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.657+0000 7ff3a0645640 10 monclient: monmap: 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:32:56.659 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 librados: starting msgr at 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 librados: starting objecter 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 Processor -- start 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 -- start start 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 librados: setting wanted keys 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 librados: calling monclient init 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient: init 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient(hunting): picked mon.noname-c con 0x7ff3980b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient(hunting): picked mon.noname-b con 0x7ff3980b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient(hunting): picked mon.noname-a con 0x7ff39814be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:32:56.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff398142050 con 0x7ff39814be90 2024-03-25T19:32:56.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3981dac50 con 0x7ff3980b4d60 2024-03-25T19:32:56.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff398140d00 con 0x7ff3980b1730 2024-03-25T19:32:56.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:56.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.658+0000 7ff3a0645640 10 monclient(hunting): authenticate will time out at 6208.817573s 2024-03-25T19:32:56.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.661+0000 7ff39e3e1640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7ff3980b1730 0x7ff3981405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:44504/0 (socket says 172.21.15.134:44504) 2024-03-25T19:32:56.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.661+0000 7ff39e3e1640 1 -- 172.21.15.134:0/2604992536 learned_addr learned my addr 172.21.15.134:0/2604992536 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:32:56.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.663+0000 7ff38effd640 10 client.?.objecter ms_handle_connect 0x7ff3980b1730 2024-03-25T19:32:56.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.664+0000 7ff38effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:56.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.664+0000 7ff38effd640 10 client.?.objecter ms_handle_connect 0x7ff39814be90 2024-03-25T19:32:56.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.664+0000 7ff38effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:56.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.670+0000 7ff38effd640 10 client.?.objecter ms_handle_connect 0x7ff3980b4d60 2024-03-25T19:32:56.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.670+0000 7ff38effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:56.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.672+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645085259 0 0) 0x7ff398142050 con 0x7ff39814be90 2024-03-25T19:32:56.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.672+0000 7ff38effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:56.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.672+0000 7ff38effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:56.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.672+0000 7ff38effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:56.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.672+0000 7ff38effd640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:32:56.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.672+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c004eb0 con 0x7ff39814be90 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.681+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646600572 0 0) 0x7ff3981dac50 con 0x7ff3980b4d60 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff398142050 con 0x7ff3980b4d60 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247296424 0 0) 0x7ff398140d00 con 0x7ff3980b1730 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:56.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:32:56.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.682+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3981dac50 con 0x7ff3980b1730 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.698+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4085515603 0 0) 0x7ff398142050 con 0x7ff3980b4d60 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.698+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff398140d00 con 0x7ff3980b4d60 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.698+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff3880018a0 con 0x7ff3980b4d60 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.698+0000 7ff38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.698+0000 7ff38effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.698+0000 7ff38effd640 10 monclient(hunting): dump: 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:56.698 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:56.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.698+0000 7ff38effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.704+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1442864939 0 0) 0x7ff3981dac50 con 0x7ff3980b1730 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.704+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff398142050 con 0x7ff3980b1730 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.704+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff394001bc0 con 0x7ff3980b1730 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.704+0000 7ff38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.704+0000 7ff38effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.704+0000 7ff38effd640 10 monclient(hunting): dump: 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:56.704 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:56.705 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:56.705 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:56.705 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:56.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.715+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 894158366 0 0) 0x7ff37c004eb0 con 0x7ff39814be90 2024-03-25T19:32:56.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.715+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3981dac50 con 0x7ff39814be90 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.715+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff390001b90 con 0x7ff39814be90 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.715+0000 7ff38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.715+0000 7ff38effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.715+0000 7ff38effd640 10 monclient(hunting): dump: 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:56.716 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2675632127 0 0) 0x7ff398142050 con 0x7ff3980b1730 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 >> v1:172.21.15.159:6789/0 conn(0x7ff3980b4d60 legacy=0x7ff3981db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 >> v1:172.21.15.134:6789/0 conn(0x7ff39814be90 legacy=0x7ff3981e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 1 monclient: found mon.c 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3981423e0 con 0x7ff3980b1730 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 10 monclient: _finish_auth 0 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:56.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff38effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 5 monclient: authenticate success, global_id 4412 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 10 monclient: _renew_subs 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39813b470 con 0x7ff3980b1730 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 10 monclient: _renew_subs 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3981dba40 con 0x7ff3980b1730 2024-03-25T19:32:56.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.717+0000 7ff3a0645640 1 librados: init done 2024-03-25T19:32:56.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.718+0000 7ff3a0645640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:56.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.718+0000 7ff3a0645640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:56.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.718+0000 7ff3a0645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:56.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.718+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3981401f0 con 0x7ff3980b1730 2024-03-25T19:32:56.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.728+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff394004a90 con 0x7ff3980b1730 2024-03-25T19:32:56.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.728+0000 7ff38effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:56.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:56.728+0000 7ff39cbde640 4 set_mon_vals no callback set 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.022+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ff394004da0 con 0x7ff3980b1730 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.022+0000 7ff38effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.022+0000 7ff38effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.022+0000 7ff38effd640 10 monclient: dump: 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:57.022 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:32:57.023 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:32:57.023 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:32:57.023 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:32:57.023 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:32:57.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.023+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7ff39401ad70 con 0x7ff3980b1730 2024-03-25T19:32:57.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.023+0000 7ff39dbe0640 10 monclient: get_auth_request con 0x7ff37c06e160 auth_method 0 2024-03-25T19:32:57.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.024+0000 7ff38effd640 10 client.4412.objecter ms_handle_connect 0x7ff37c06e160 2024-03-25T19:32:57.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7ff394054940 con 0x7ff3980b1730 2024-03-25T19:32:57.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 10 client.4412.objecter ms_dispatch 0x7ff398000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:57.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:57.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:57.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T19:32:57.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff394050450 con 0x7ff3980b1730 2024-03-25T19:32:57.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:57.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.029+0000 7ff38effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:57.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.285+0000 7ff3a0645640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:57.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.285+0000 7ff3a0645640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:57.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.285+0000 7ff3a0645640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:32:57.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.285+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff398140e90 con 0x7ff3980b1730 2024-03-25T19:32:57.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.332+0000 7ff38effd640 1 -- 172.21.15.134:0/2604992536 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7ff394052380 con 0x7ff3980b1730 2024-03-25T19:32:57.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.332+0000 7ff38effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:57.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.332+0000 7ff38effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:57.332 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:32:57.332 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 10 librados: watch_flush enter 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 10 librados: watch_flush exit 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 10 client.4412.objecter successfully canceled tick 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 >> v1:172.21.15.159:6800/1875874887 conn(0x7ff37c06e160 legacy=0x7ff37c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 10 monclient: shutdown 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 >> v1:172.21.15.134:6790/0 conn(0x7ff3980b1730 legacy=0x7ff3981405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 shutdown_connections 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.334+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 >> 172.21.15.134:0/2604992536 conn(0x7ff3980617a0 msgr2=0x7ff3980b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.335+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 shutdown_connections 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.335+0000 7ff3a0645640 1 -- 172.21.15.134:0/2604992536 wait complete. 2024-03-25T19:32:57.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:32:57.335+0000 7ff3a0645640 1 librados: shutdown 2024-03-25T19:32:57.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:04.153 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.269+0000 7f756356d640 10 monclient: get_monmap_and_config 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.269+0000 7f756356d640 10 monclient: build_initial_monmap 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.269+0000 7f756356d640 10 monclient: monmap: 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:04.270305+0000 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:04.270305+0000 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:04.269 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:04.270 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:04.270 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:04.270 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.270+0000 7f756356d640 1 Processor -- start 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.270+0000 7f756356d640 1 -- start start 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.270+0000 7f756356d640 10 monclient: init 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 10 monclient(hunting): picked mon.noname-c con 0x7f755c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 10 monclient(hunting): picked mon.noname-b con 0x7f755c0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 10 monclient(hunting): picked mon.noname-a con 0x7f755c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c0af940 con 0x7f755c14be90 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c0b0a70 con 0x7f755c0b4d60 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c0b0d50 con 0x7f755c0b1730 2024-03-25T19:33:04.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:04.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.271+0000 7f756356d640 10 monclient(hunting): authenticate will time out at 6216.429929s 2024-03-25T19:33:04.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.277+0000 7f7561309640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f755c0b4d60 0x7f755c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:37368/0 (socket says 172.21.15.134:37368) 2024-03-25T19:33:04.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.277+0000 7f7561309640 1 -- 172.21.15.134:0/13963488 learned_addr learned my addr 172.21.15.134:0/13963488 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:04.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.292+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483845458 0 0) 0x7f755c0af940 con 0x7f755c14be90 2024-03-25T19:33:04.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.292+0000 7f7553fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:04.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.292+0000 7f7553fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.292+0000 7f7553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.292+0000 7f7553fff640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.292+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548003520 con 0x7f755c14be90 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.293+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100029473 0 0) 0x7f755c0b0a70 con 0x7f755c0b4d60 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.293+0000 7f7553fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.293+0000 7f7553fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.293+0000 7f7553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.293+0000 7f7553fff640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.293+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c0af940 con 0x7f755c0b4d60 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.299+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350720089 0 0) 0x7f755c0af940 con 0x7f755c0b4d60 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 >> v1:172.21.15.134:6790/0 conn(0x7f755c0b1730 legacy=0x7f755c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 >> v1:172.21.15.134:6789/0 conn(0x7f755c14be90 legacy=0x7f755c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 1 monclient: found mon.noname-c 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0b1030 con 0x7f755c0b4d60 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 10 monclient: _finish_auth 0 2024-03-25T19:33:04.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7558001670 con 0x7f755c0b4d60 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f7553fff640 10 monclient: dump: 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f756356d640 5 monclient: authenticate success, global_id 4411 2024-03-25T19:33:04.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.300+0000 7f756356d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:04.305 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75580017d0 con 0x7f755c0b4d60 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f7553fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f7553fff640 1 -- 172.21.15.134:0/13963488 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7558001ba0 con 0x7f755c0b4d60 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f756356d640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f756356d640 4 set_mon_vals no callback set 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f7553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f7553fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.305+0000 7f7553fff640 10 monclient: dump: 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:04.306 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:04.307 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.306+0000 7f756356d640 10 monclient: shutdown 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.306+0000 7f756356d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.306+0000 7f756356d640 1 -- 172.21.15.134:0/13963488 >> v1:172.21.15.159:6789/0 conn(0x7f755c0b4d60 legacy=0x7f755c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.306+0000 7f756356d640 1 -- 172.21.15.134:0/13963488 shutdown_connections 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.306+0000 7f756356d640 1 -- 172.21.15.134:0/13963488 >> 172.21.15.134:0/13963488 conn(0x7f755c0617a0 msgr2=0x7f755c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.306+0000 7f756356d640 1 -- 172.21.15.134:0/13963488 shutdown_connections 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.307+0000 7f756356d640 1 -- 172.21.15.134:0/13963488 wait complete. 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 10 monclient: build_initial_monmap 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 10 monclient: monmap: 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:04.308 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 1 librados: starting msgr at 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 1 librados: starting objecter 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 1 Processor -- start 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 1 -- start start 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 1 librados: setting wanted keys 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 1 librados: calling monclient init 2024-03-25T19:33:04.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.308+0000 7f756356d640 10 monclient: init 2024-03-25T19:33:04.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:04.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:04.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 10 monclient(hunting): picked mon.noname-b con 0x7f755c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:04.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 10 monclient(hunting): picked mon.noname-c con 0x7f755c0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:04.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 10 monclient(hunting): picked mon.noname-a con 0x7f755c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:04.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c13b000 con 0x7f755c14be90 2024-03-25T19:33:04.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c1d7b10 con 0x7f755c0b1730 2024-03-25T19:33:04.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c1d77a0 con 0x7f755c0b4d60 2024-03-25T19:33:04.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:04.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.309+0000 7f756356d640 10 monclient(hunting): authenticate will time out at 6216.468039s 2024-03-25T19:33:04.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.318+0000 7f7561b0a640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f755c14be90 0x7f755c1d8030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:59470/0 (socket says 172.21.15.134:59470) 2024-03-25T19:33:04.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.318+0000 7f7561b0a640 1 -- 172.21.15.134:0/2873324179 learned_addr learned my addr 172.21.15.134:0/2873324179 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:04.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.321+0000 7f7551ffb640 10 client.?.objecter ms_handle_connect 0x7f755c0b1730 2024-03-25T19:33:04.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.321+0000 7f7551ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:04.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.321+0000 7f7551ffb640 10 client.?.objecter ms_handle_connect 0x7f755c14be90 2024-03-25T19:33:04.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.321+0000 7f7551ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:04.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.330+0000 7f7551ffb640 10 client.?.objecter ms_handle_connect 0x7f755c0b4d60 2024-03-25T19:33:04.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.330+0000 7f7551ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:04.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.331+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645100974 0 0) 0x7f755c1d7b10 con 0x7f755c0b1730 2024-03-25T19:33:04.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.331+0000 7f7551ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:04.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.331+0000 7f7551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:04.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.331+0000 7f7551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:04.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.331+0000 7f7551ffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:33:04.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.331+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7540004eb0 con 0x7f755c0b1730 2024-03-25T19:33:04.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.339+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712169864 0 0) 0x7f755c13b000 con 0x7f755c14be90 2024-03-25T19:33:04.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.339+0000 7f7551ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:04.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.339+0000 7f7551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:04.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.339+0000 7f7551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:04.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.339+0000 7f7551ffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:33:04.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.339+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c1d7b10 con 0x7f755c14be90 2024-03-25T19:33:04.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.350+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3003489770 0 0) 0x7f7540004eb0 con 0x7f755c0b1730 2024-03-25T19:33:04.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.350+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f755c13b000 con 0x7f755c0b1730 2024-03-25T19:33:04.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.350+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7558001ae0 con 0x7f755c0b1730 2024-03-25T19:33:04.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.350+0000 7f7551ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:04.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.350+0000 7f7551ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:04.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.350+0000 7f7551ffb640 10 monclient(hunting): dump: 2024-03-25T19:33:04.350 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:04.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.350+0000 7f7551ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:04.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.354+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3223052988 0 0) 0x7f755c1d7b10 con 0x7f755c14be90 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.354+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7540004eb0 con 0x7f755c14be90 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.354+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7554001bf0 con 0x7f755c14be90 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.354+0000 7f7551ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.354+0000 7f7551ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.354+0000 7f7551ffb640 10 monclient(hunting): dump: 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:04.354 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:04.355 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:04.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.357+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406358302 0 0) 0x7f755c1d77a0 con 0x7f755c0b4d60 2024-03-25T19:33:04.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.357+0000 7f7551ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:04.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.357+0000 7f7551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:04.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.357+0000 7f7551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:04.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.357+0000 7f7551ffb640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:33:04.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.357+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c1d7b10 con 0x7f755c0b4d60 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 515479280 0 0) 0x7f755c13b000 con 0x7f755c0b1730 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 >> v1:172.21.15.134:6790/0 conn(0x7f755c0b4d60 legacy=0x7f755c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 >> v1:172.21.15.134:6789/0 conn(0x7f755c14be90 legacy=0x7f755c1d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 1 monclient: found mon.b 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c142fe0 con 0x7f755c0b1730 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:04.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f7551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 10 monclient: _renew_subs 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c1431f0 con 0x7f755c0b1730 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 10 monclient: _renew_subs 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c13bf90 con 0x7f755c0b1730 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.360+0000 7f756356d640 1 librados: init done 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.361+0000 7f756356d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.361+0000 7f756356d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.361+0000 7f756356d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:04.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.361+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c1d7930 con 0x7f755c0b1730 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.631+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7558004a90 con 0x7f755c0b1730 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.631+0000 7f7551ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.631+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7558004da0 con 0x7f755c0b1730 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.631+0000 7f7551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.631+0000 7f7551ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.631+0000 7f7551ffb640 10 monclient: dump: 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:04.631 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:04.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.631+0000 7f75537fe640 4 set_mon_vals no callback set 2024-03-25T19:33:04.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.632+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f755801ad70 con 0x7f755c0b1730 2024-03-25T19:33:04.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.632+0000 7f7560b08640 10 monclient: get_auth_request con 0x7f754006df00 auth_method 0 2024-03-25T19:33:04.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.633+0000 7f7551ffb640 10 client.4414.objecter ms_handle_connect 0x7f754006df00 2024-03-25T19:33:04.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f75580548f0 con 0x7f755c0b1730 2024-03-25T19:33:04.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 10 client.4414.objecter ms_dispatch 0x7f755c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:04.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:04.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:04.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T19:33:04.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7558050230 con 0x7f755c0b1730 2024-03-25T19:33:04.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:04.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.636+0000 7f7551ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:04.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.849+0000 7f756356d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:04.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.849+0000 7f756356d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:04.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.849+0000 7f756356d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:04.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.849+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f755c0627b0 con 0x7f755c0b1730 2024-03-25T19:33:04.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.959+0000 7f7551ffb640 1 -- 172.21.15.134:0/2873324179 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f7558052330 con 0x7f755c0b1730 2024-03-25T19:33:04.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.959+0000 7f7551ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:04.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.959+0000 7f7551ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:04.960 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:04.960 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:04.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 10 librados: watch_flush enter 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 10 librados: watch_flush exit 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 10 client.4414.objecter successfully canceled tick 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 >> v1:172.21.15.159:6800/1875874887 conn(0x7f754006df00 legacy=0x7f75400703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 10 monclient: shutdown 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 >> v1:172.21.15.159:6789/0 conn(0x7f755c0b1730 legacy=0x7f755c13b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 shutdown_connections 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 >> 172.21.15.134:0/2873324179 conn(0x7f755c0617a0 msgr2=0x7f755c0b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 shutdown_connections 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 1 -- 172.21.15.134:0/2873324179 wait complete. 2024-03-25T19:33:04.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:04.962+0000 7f756356d640 1 librados: shutdown 2024-03-25T19:33:04.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:05.256 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:09.929 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.312 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:09.984 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.341 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:10.013 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.385 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:10.032 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.385 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:10.058 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.392 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:10.039 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.397 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:10.070 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.415 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:10.062 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:05.428 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:10.101 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:11.780 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.899+0000 7fc37caff640 10 monclient: get_monmap_and_config 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.899+0000 7fc37caff640 10 monclient: build_initial_monmap 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.899+0000 7fc37caff640 10 monclient: monmap: 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:11.900447+0000 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:11.900447+0000 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:11.899 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:11.900 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:11.900 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:11.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.900+0000 7fc37caff640 1 Processor -- start 2024-03-25T19:33:11.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.900+0000 7fc37caff640 1 -- start start 2024-03-25T19:33:11.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.900+0000 7fc37caff640 10 monclient: init 2024-03-25T19:33:11.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:11.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:11.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 10 monclient(hunting): picked mon.noname-a con 0x7fc3780a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:11.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 10 monclient(hunting): picked mon.noname-b con 0x7fc37813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:11.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 10 monclient(hunting): picked mon.noname-c con 0x7fc3781430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:11.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc378142520 con 0x7fc3780a98f0 2024-03-25T19:33:11.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780b28a0 con 0x7fc3781430c0 2024-03-25T19:33:11.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780b2a30 con 0x7fc37813f710 2024-03-25T19:33:11.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:11.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.901+0000 7fc37caff640 10 monclient(hunting): authenticate will time out at 6224.059964s 2024-03-25T19:33:11.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.907+0000 7fc36dd9b640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc37813f710 0x7fc3781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:52718/0 (socket says 172.21.15.134:52718) 2024-03-25T19:33:11.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.907+0000 7fc36dd9b640 1 -- 172.21.15.134:0/2489327494 learned_addr learned my addr 172.21.15.134:0/2489327494 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:11.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.918+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185531825 0 0) 0x7fc3780b2a30 con 0x7fc37813f710 2024-03-25T19:33:11.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.918+0000 7fc375d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:11.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.918+0000 7fc375d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:11.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.918+0000 7fc375d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:11.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.918+0000 7fc375d9b640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:33:11.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.918+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc364003520 con 0x7fc37813f710 2024-03-25T19:33:11.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3631482523 0 0) 0x7fc364003520 con 0x7fc37813f710 2024-03-25T19:33:11.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 >> v1:172.21.15.159:6789/0 conn(0x7fc3781430c0 legacy=0x7fc37814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:11.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 >> v1:172.21.15.134:6789/0 conn(0x7fc3780a98f0 legacy=0x7fc3780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:11.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:33:11.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 1 monclient: found mon.noname-b 2024-03-25T19:33:11.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780b44c0 con 0x7fc37813f710 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 10 monclient: _finish_auth 0 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc360001690 con 0x7fc37813f710 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc37caff640 5 monclient: authenticate success, global_id 4421 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc37caff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc375d9b640 10 monclient: dump: 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:11.932 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:11.933 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:11.933 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:11.933 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:11.933 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:11.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.931+0000 7fc37caff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:11.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc360001860 con 0x7fc37813f710 2024-03-25T19:33:11.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc375d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:11.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc375d9b640 1 -- 172.21.15.134:0/2489327494 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc360001cb0 con 0x7fc37813f710 2024-03-25T19:33:11.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc37caff640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:11.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc37caff640 4 set_mon_vals no callback set 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc375d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc375d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.945+0000 7fc375d9b640 10 monclient: dump: 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:11.946 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:11.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.946+0000 7fc37caff640 10 monclient: shutdown 2024-03-25T19:33:11.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.946+0000 7fc37caff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:11.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.946+0000 7fc37caff640 1 -- 172.21.15.134:0/2489327494 >> v1:172.21.15.134:6790/0 conn(0x7fc37813f710 legacy=0x7fc3781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:11.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.946+0000 7fc37caff640 1 -- 172.21.15.134:0/2489327494 shutdown_connections 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.946+0000 7fc37caff640 1 -- 172.21.15.134:0/2489327494 >> 172.21.15.134:0/2489327494 conn(0x7fc3780b7ea0 msgr2=0x7fc3780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.946+0000 7fc37caff640 1 -- 172.21.15.134:0/2489327494 shutdown_connections 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.946+0000 7fc37caff640 1 -- 172.21.15.134:0/2489327494 wait complete. 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.947+0000 7fc37caff640 10 monclient: build_initial_monmap 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.947+0000 7fc37caff640 10 monclient: monmap: 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:11.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.947+0000 7fc37caff640 1 librados: starting msgr at 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.947+0000 7fc37caff640 1 librados: starting objecter 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 1 Processor -- start 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 1 -- start start 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 1 librados: setting wanted keys 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 1 librados: calling monclient init 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient: init 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient(hunting): picked mon.noname-b con 0x7fc3780a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient(hunting): picked mon.noname-c con 0x7fc37813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient(hunting): picked mon.noname-a con 0x7fc3781430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3781394f0 con 0x7fc3781430c0 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3781d75f0 con 0x7fc3780a98f0 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3781d8190 con 0x7fc37813f710 2024-03-25T19:33:11.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:11.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.948+0000 7fc37caff640 10 monclient(hunting): authenticate will time out at 6224.107385s 2024-03-25T19:33:11.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.951+0000 7fc37659c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc3780a98f0 0x7fc3781d7a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:55964/0 (socket says 172.21.15.134:55964) 2024-03-25T19:33:11.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.951+0000 7fc37659c640 1 -- 172.21.15.134:0/1902689153 learned_addr learned my addr 172.21.15.134:0/1902689153 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:11.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.958+0000 7fc36f7fe640 10 client.?.objecter ms_handle_connect 0x7fc3780a98f0 2024-03-25T19:33:11.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.958+0000 7fc36f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:11.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.959+0000 7fc36f7fe640 10 client.?.objecter ms_handle_connect 0x7fc3781430c0 2024-03-25T19:33:11.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.959+0000 7fc36f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:11.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.967+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440531092 0 0) 0x7fc3781394f0 con 0x7fc3781430c0 2024-03-25T19:33:11.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.967+0000 7fc36f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:11.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.967+0000 7fc36f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:11.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.967+0000 7fc36f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:11.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.967+0000 7fc36f7fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:33:11.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.967+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc358004eb0 con 0x7fc3781430c0 2024-03-25T19:33:11.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.968+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482133461 0 0) 0x7fc3781d75f0 con 0x7fc3780a98f0 2024-03-25T19:33:11.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.968+0000 7fc36f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:11.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.968+0000 7fc36f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:11.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.968+0000 7fc36f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:11.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.968+0000 7fc36f7fe640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:33:11.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.968+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3781394f0 con 0x7fc3780a98f0 2024-03-25T19:33:11.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.977+0000 7fc36f7fe640 10 client.?.objecter ms_handle_connect 0x7fc37813f710 2024-03-25T19:33:11.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.977+0000 7fc36f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:11.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.979+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500362901 0 0) 0x7fc358004eb0 con 0x7fc3781430c0 2024-03-25T19:33:11.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.979+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3781d75f0 con 0x7fc3781430c0 2024-03-25T19:33:11.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.979+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc370001ad0 con 0x7fc3781430c0 2024-03-25T19:33:11.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.979+0000 7fc36f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:11.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.979+0000 7fc36f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.979+0000 7fc36f7fe640 10 monclient(hunting): dump: 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:11.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.979+0000 7fc36f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.983+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486686848 0 0) 0x7fc3781394f0 con 0x7fc3780a98f0 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.983+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc358004eb0 con 0x7fc3780a98f0 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.983+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc368004040 con 0x7fc3780a98f0 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.983+0000 7fc36f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.983+0000 7fc36f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.983+0000 7fc36f7fe640 10 monclient(hunting): dump: 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:11.983 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:11.984 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3621398313 0 0) 0x7fc358004eb0 con 0x7fc3780a98f0 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 >> v1:172.21.15.134:6790/0 conn(0x7fc37813f710 legacy=0x7fc378138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 >> v1:172.21.15.134:6789/0 conn(0x7fc3781430c0 legacy=0x7fc378133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 1 monclient: found mon.b 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc378138da0 con 0x7fc3780a98f0 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:11.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc36f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 5 monclient: authenticate success, global_id 4417 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 10 monclient: _renew_subs 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc378139740 con 0x7fc3780a98f0 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 10 monclient: _renew_subs 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3781d7280 con 0x7fc3780a98f0 2024-03-25T19:33:11.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.989+0000 7fc37caff640 1 librados: init done 2024-03-25T19:33:11.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.990+0000 7fc37caff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:11.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.990+0000 7fc37caff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:11.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.990+0000 7fc37caff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:11.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:11.990+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37814f2d0 con 0x7fc3780a98f0 2024-03-25T19:33:12.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.259+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc368000ed0 con 0x7fc3780a98f0 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.259+0000 7fc36f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.259+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc368001220 con 0x7fc3780a98f0 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.259+0000 7fc36f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.259+0000 7fc36f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.259+0000 7fc36f7fe640 10 monclient: dump: 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:12.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.259+0000 7fc37559a640 4 set_mon_vals no callback set 2024-03-25T19:33:12.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.260+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fc368001440 con 0x7fc3780a98f0 2024-03-25T19:33:12.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.261+0000 7fc36dd9b640 10 monclient: get_auth_request con 0x7fc35806df20 auth_method 0 2024-03-25T19:33:12.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.261+0000 7fc36f7fe640 10 client.4417.objecter ms_handle_connect 0x7fc35806df20 2024-03-25T19:33:12.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.267+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fc368054730 con 0x7fc3780a98f0 2024-03-25T19:33:12.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.267+0000 7fc36f7fe640 10 client.4417.objecter ms_dispatch 0x7fc378000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:12.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.267+0000 7fc36f7fe640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:12.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.267+0000 7fc36f7fe640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:12.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.268+0000 7fc36f7fe640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T19:33:12.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.268+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc368050240 con 0x7fc3780a98f0 2024-03-25T19:33:12.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.268+0000 7fc36f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:12.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.268+0000 7fc36f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:12.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.495+0000 7fc37caff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:12.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.495+0000 7fc37caff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:12.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.495+0000 7fc37caff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:12.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.495+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3780627b0 con 0x7fc3780a98f0 2024-03-25T19:33:12.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.521+0000 7fc36f7fe640 1 -- 172.21.15.134:0/1902689153 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fc368052170 con 0x7fc3780a98f0 2024-03-25T19:33:12.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.521+0000 7fc36f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:12.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.521+0000 7fc36f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:12.521 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:12.521 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.523+0000 7fc37caff640 10 librados: watch_flush enter 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.523+0000 7fc37caff640 10 librados: watch_flush exit 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.523+0000 7fc37caff640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.523+0000 7fc37caff640 10 client.4417.objecter successfully canceled tick 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 >> v1:172.21.15.159:6800/1875874887 conn(0x7fc35806df20 legacy=0x7fc3580703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 10 monclient: shutdown 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:12.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 >> v1:172.21.15.159:6789/0 conn(0x7fc3780a98f0 legacy=0x7fc3781d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:12.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 shutdown_connections 2024-03-25T19:33:12.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 >> 172.21.15.134:0/1902689153 conn(0x7fc3780b7ea0 msgr2=0x7fc378074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:12.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 shutdown_connections 2024-03-25T19:33:12.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 1 -- 172.21.15.134:0/1902689153 wait complete. 2024-03-25T19:33:12.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:12.524+0000 7fc37caff640 1 librados: shutdown 2024-03-25T19:33:12.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:19.342 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:19.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.458+0000 7f48c2193640 10 monclient: get_monmap_and_config 2024-03-25T19:33:19.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.458+0000 7f48c2193640 10 monclient: build_initial_monmap 2024-03-25T19:33:19.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.458+0000 7f48c2193640 10 monclient: monmap: 2024-03-25T19:33:19.458 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:19.459879+0000 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:19.459879+0000 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:19.459 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:19.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.459+0000 7f48c2193640 1 Processor -- start 2024-03-25T19:33:19.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.459+0000 7f48c2193640 1 -- start start 2024-03-25T19:33:19.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.459+0000 7f48c2193640 10 monclient: init 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 10 monclient(hunting): picked mon.noname-b con 0x7f48bc0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 10 monclient(hunting): picked mon.noname-c con 0x7f48bc0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 10 monclient(hunting): picked mon.noname-a con 0x7f48bc14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0af940 con 0x7f48bc14be90 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b0a70 con 0x7f48bc0b1730 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b0d50 con 0x7f48bc0b4d60 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:19.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.460+0000 7f48c2193640 10 monclient(hunting): authenticate will time out at 6231.618943s 2024-03-25T19:33:19.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.469+0000 7f48baffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f48bc0b1730 0x7f48bc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:60564/0 (socket says 172.21.15.134:60564) 2024-03-25T19:33:19.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.469+0000 7f48baffd640 1 -- 172.21.15.134:0/1713425421 learned_addr learned my addr 172.21.15.134:0/1713425421 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:19.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.479+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106976322 0 0) 0x7f48bc0b0a70 con 0x7f48bc0b1730 2024-03-25T19:33:19.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.479+0000 7f48ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:19.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.479+0000 7f48ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:19.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.479+0000 7f48ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:19.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.479+0000 7f48ba7fc640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:33:19.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.479+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a4003520 con 0x7f48bc0b1730 2024-03-25T19:33:19.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.485+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199016803 0 0) 0x7f48bc0b0d50 con 0x7f48bc0b4d60 2024-03-25T19:33:19.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.485+0000 7f48ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:19.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.485+0000 7f48ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:19.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.485+0000 7f48ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:19.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.485+0000 7f48ba7fc640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:33:19.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.485+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48bc0b0a70 con 0x7f48bc0b4d60 2024-03-25T19:33:19.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.486+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674367459 0 0) 0x7f48bc0af940 con 0x7f48bc14be90 2024-03-25T19:33:19.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.486+0000 7f48ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:19.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.486+0000 7f48ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:19.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.486+0000 7f48ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:19.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.486+0000 7f48ba7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:33:19.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.486+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48bc0b0d50 con 0x7f48bc14be90 2024-03-25T19:33:19.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3856325049 0 0) 0x7f48bc0b0a70 con 0x7f48bc0b4d60 2024-03-25T19:33:19.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 >> v1:172.21.15.159:6789/0 conn(0x7f48bc0b1730 legacy=0x7f48bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:19.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 >> v1:172.21.15.134:6789/0 conn(0x7f48bc14be90 legacy=0x7f48bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:19.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:19.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 1 monclient: found mon.noname-b 2024-03-25T19:33:19.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0b1030 con 0x7f48bc0b4d60 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f48ac001670 con 0x7f48bc0b4d60 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48c2193640 5 monclient: authenticate success, global_id 4427 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48c2193640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48ba7fc640 10 monclient: dump: 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:19.501 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:19.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.500+0000 7f48c2193640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:19.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48ac0017d0 con 0x7f48bc0b4d60 2024-03-25T19:33:19.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:19.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48ba7fc640 1 -- 172.21.15.134:0/1713425421 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f48ac001be0 con 0x7f48bc0b4d60 2024-03-25T19:33:19.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:19.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:19.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48ba7fc640 10 monclient: dump: 2024-03-25T19:33:19.509 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48c2193640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:19.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.509+0000 7f48c2193640 4 set_mon_vals no callback set 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.510+0000 7f48c2193640 10 monclient: shutdown 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.510+0000 7f48c2193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.510+0000 7f48c2193640 1 -- 172.21.15.134:0/1713425421 >> v1:172.21.15.134:6790/0 conn(0x7f48bc0b4d60 legacy=0x7f48bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.510+0000 7f48c2193640 1 -- 172.21.15.134:0/1713425421 shutdown_connections 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.510+0000 7f48c2193640 1 -- 172.21.15.134:0/1713425421 >> 172.21.15.134:0/1713425421 conn(0x7f48bc0617a0 msgr2=0x7f48bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.510+0000 7f48c2193640 1 -- 172.21.15.134:0/1713425421 shutdown_connections 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.510+0000 7f48c2193640 1 -- 172.21.15.134:0/1713425421 wait complete. 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.511+0000 7f48c2193640 10 monclient: build_initial_monmap 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.511+0000 7f48c2193640 10 monclient: monmap: 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:19.512 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.511+0000 7f48c2193640 1 librados: starting msgr at 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.511+0000 7f48c2193640 1 librados: starting objecter 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.511+0000 7f48c2193640 1 Processor -- start 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 1 -- start start 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 1 librados: setting wanted keys 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 1 librados: calling monclient init 2024-03-25T19:33:19.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient: init 2024-03-25T19:33:19.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:19.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:19.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient(hunting): picked mon.noname-a con 0x7f48bc0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:19.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient(hunting): picked mon.noname-b con 0x7f48bc0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:19.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient(hunting): picked mon.noname-c con 0x7f48bc14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:19.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc1db0d0 con 0x7f48bc0b1730 2024-03-25T19:33:19.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc1dbd00 con 0x7f48bc0b4d60 2024-03-25T19:33:19.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc1dbe90 con 0x7f48bc14be90 2024-03-25T19:33:19.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:19.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.512+0000 7f48c2193640 10 monclient(hunting): authenticate will time out at 6231.671460s 2024-03-25T19:33:19.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.526+0000 7f48bbfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f48bc14be90 0x7f48bc1e32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:57356/0 (socket says 172.21.15.134:57356) 2024-03-25T19:33:19.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.526+0000 7f48bbfff640 1 -- 172.21.15.134:0/3775777736 learned_addr learned my addr 172.21.15.134:0/3775777736 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:19.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.528+0000 7f48a3fff640 10 client.?.objecter ms_handle_connect 0x7f48bc14be90 2024-03-25T19:33:19.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.528+0000 7f48a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:19.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.529+0000 7f48a3fff640 10 client.?.objecter ms_handle_connect 0x7f48bc0b4d60 2024-03-25T19:33:19.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.529+0000 7f48a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:19.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.540+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549863119 0 0) 0x7f48bc1dbd00 con 0x7f48bc0b4d60 2024-03-25T19:33:19.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.540+0000 7f48a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:19.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.540+0000 7f48a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:19.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.540+0000 7f48a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:19.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.540+0000 7f48a3fff640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:33:19.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.540+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a8004eb0 con 0x7f48bc0b4d60 2024-03-25T19:33:19.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.542+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644526801 0 0) 0x7f48bc0b0d50 con 0x7f48bc14be90 2024-03-25T19:33:19.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.542+0000 7f48a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:19.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.542+0000 7f48a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:19.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.542+0000 7f48a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:19.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.542+0000 7f48a3fff640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:33:19.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.542+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48bc1dbd00 con 0x7f48bc14be90 2024-03-25T19:33:19.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.547+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3066566051 0 0) 0x7f48a8004eb0 con 0x7f48bc0b4d60 2024-03-25T19:33:19.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.547+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48bc0b0d50 con 0x7f48bc0b4d60 2024-03-25T19:33:19.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.547+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f48b0001950 con 0x7f48bc0b4d60 2024-03-25T19:33:19.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.547+0000 7f48a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:19.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.547+0000 7f48a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:19.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.547+0000 7f48a3fff640 10 monclient(hunting): dump: 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:19.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.547+0000 7f48a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.555+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1804251267 0 0) 0x7f48bc1dbd00 con 0x7f48bc14be90 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.555+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48a8004eb0 con 0x7f48bc14be90 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.555+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f48b4001ab0 con 0x7f48bc14be90 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.555+0000 7f48a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.555+0000 7f48a3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.555+0000 7f48a3fff640 10 monclient(hunting): dump: 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.555 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.556 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:19.556 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:19.556 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:19.556 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:19.556 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:19.556 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:19.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.557+0000 7f48a3fff640 10 client.?.objecter ms_handle_connect 0x7f48bc0b1730 2024-03-25T19:33:19.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.557+0000 7f48a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:19.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 383778262 0 0) 0x7f48a8004eb0 con 0x7f48bc14be90 2024-03-25T19:33:19.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 >> v1:172.21.15.159:6789/0 conn(0x7f48bc0b4d60 legacy=0x7f48bc13b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:19.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 >> v1:172.21.15.134:6789/0 conn(0x7f48bc0b1730 legacy=0x7f48bc1db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:19.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:19.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 1 monclient: found mon.c 2024-03-25T19:33:19.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc13be30 con 0x7f48bc14be90 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 5 monclient: authenticate success, global_id 4430 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 10 monclient: _renew_subs 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc13c080 con 0x7f48bc14be90 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 10 monclient: _renew_subs 2024-03-25T19:33:19.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:19.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc1399b0 con 0x7f48bc14be90 2024-03-25T19:33:19.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.561+0000 7f48c2193640 1 librados: init done 2024-03-25T19:33:19.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.562+0000 7f48c2193640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:19.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.562+0000 7f48c2193640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:19.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.562+0000 7f48c2193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:19.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.562+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc139c00 con 0x7f48bc14be90 2024-03-25T19:33:19.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.853+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48b4001fa0 con 0x7f48bc14be90 2024-03-25T19:33:19.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.853+0000 7f48a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:19.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.853+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f48b4003b30 con 0x7f48bc14be90 2024-03-25T19:33:19.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.853+0000 7f48a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.853+0000 7f48a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.853+0000 7f48a3fff640 10 monclient: dump: 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:19.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.853+0000 7f48b9ffb640 4 set_mon_vals no callback set 2024-03-25T19:33:19.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.854+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f48b4003d50 con 0x7f48bc14be90 2024-03-25T19:33:19.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.854+0000 7f48bb7fe640 10 monclient: get_auth_request con 0x7f48a806dec0 auth_method 0 2024-03-25T19:33:19.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.855+0000 7f48a3fff640 10 client.4430.objecter ms_handle_connect 0x7f48a806dec0 2024-03-25T19:33:19.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.860+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f48b4054850 con 0x7f48bc14be90 2024-03-25T19:33:19.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.860+0000 7f48a3fff640 10 client.4430.objecter ms_dispatch 0x7f48bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:19.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.860+0000 7f48a3fff640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:19.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.860+0000 7f48a3fff640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:19.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.860+0000 7f48a3fff640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-25T19:33:19.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.861+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48b4050360 con 0x7f48bc14be90 2024-03-25T19:33:19.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.861+0000 7f48a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:19.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:19.861+0000 7f48a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:20.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.106+0000 7f48c2193640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:20.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.106+0000 7f48c2193640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:20.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.106+0000 7f48c2193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:20.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.106+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48bc1401f0 con 0x7f48bc14be90 2024-03-25T19:33:20.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.132+0000 7f48a3fff640 1 -- 172.21.15.134:0/3775777736 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f48b4052290 con 0x7f48bc14be90 2024-03-25T19:33:20.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.132+0000 7f48a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:20.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.132+0000 7f48a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:20.132 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:20.132 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 10 librados: watch_flush enter 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 10 librados: watch_flush exit 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 10 client.4430.objecter successfully canceled tick 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 >> v1:172.21.15.159:6800/1875874887 conn(0x7f48a806dec0 legacy=0x7f48a8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 10 monclient: shutdown 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 >> v1:172.21.15.134:6790/0 conn(0x7f48bc14be90 legacy=0x7f48bc1e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 shutdown_connections 2024-03-25T19:33:20.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.134+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 >> 172.21.15.134:0/3775777736 conn(0x7f48bc0617a0 msgr2=0x7f48bc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:20.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.135+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 shutdown_connections 2024-03-25T19:33:20.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.135+0000 7f48c2193640 1 -- 172.21.15.134:0/3775777736 wait complete. 2024-03-25T19:33:20.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:20.135+0000 7f48c2193640 1 librados: shutdown 2024-03-25T19:33:20.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:25.345 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:30.017 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.354 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:30.026 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.392 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:30.065 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.416 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:30.088 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.419 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:30.091 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.425 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:30.072 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.430 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:30.077 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.447 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:30.119 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:25.459 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:30.106 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:26.953 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:27.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.068+0000 7f110c155640 10 monclient: get_monmap_and_config 2024-03-25T19:33:27.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.068+0000 7f110c155640 10 monclient: build_initial_monmap 2024-03-25T19:33:27.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.068+0000 7f110c155640 10 monclient: monmap: 2024-03-25T19:33:27.068 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:27.068 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:27.069 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:27.069071+0000 2024-03-25T19:33:27.069 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:27.069071+0000 2024-03-25T19:33:27.069 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:27.069 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:27.069 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:27.069 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:27.070 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:27.070 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:27.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.068+0000 7f110c155640 1 Processor -- start 2024-03-25T19:33:27.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 1 -- start start 2024-03-25T19:33:27.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient: init 2024-03-25T19:33:27.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:27.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:27.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient(hunting): picked mon.noname-c con 0x7f11040b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:27.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient(hunting): picked mon.noname-b con 0x7f11040b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:27.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient(hunting): picked mon.noname-a con 0x7f110414be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:27.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040af940 con 0x7f110414be90 2024-03-25T19:33:27.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040b0a70 con 0x7f11040b4d60 2024-03-25T19:33:27.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040b0d50 con 0x7f11040b1730 2024-03-25T19:33:27.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:27.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.069+0000 7f110c155640 10 monclient(hunting): authenticate will time out at 6239.228568s 2024-03-25T19:33:27.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.076+0000 7f110a6f2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f110414be90 0x7f110414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49164/0 (socket says 172.21.15.134:49164) 2024-03-25T19:33:27.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.076+0000 7f110a6f2640 1 -- 172.21.15.134:0/1000542165 learned_addr learned my addr 172.21.15.134:0/1000542165 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:27.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.086+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125934575 0 0) 0x7f11040af940 con 0x7f110414be90 2024-03-25T19:33:27.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.086+0000 7f1108eef640 10 monclient(hunting): _negotiate 2024-03-25T19:33:27.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.086+0000 7f1108eef640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:27.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.086+0000 7f1108eef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:27.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.086+0000 7f1108eef640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:33:27.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.086+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f0003520 con 0x7f110414be90 2024-03-25T19:33:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.095+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399207897 0 0) 0x7f11040b0d50 con 0x7f11040b1730 2024-03-25T19:33:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.095+0000 7f1108eef640 10 monclient(hunting): _negotiate 2024-03-25T19:33:27.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.095+0000 7f1108eef640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:27.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.095+0000 7f1108eef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:27.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.095+0000 7f1108eef640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:33:27.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.095+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11040af940 con 0x7f11040b1730 2024-03-25T19:33:27.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.097+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204741286 0 0) 0x7f11040b0a70 con 0x7f11040b4d60 2024-03-25T19:33:27.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.097+0000 7f1108eef640 10 monclient(hunting): _negotiate 2024-03-25T19:33:27.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.097+0000 7f1108eef640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:27.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.097+0000 7f1108eef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:27.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.097+0000 7f1108eef640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:33:27.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.097+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11040b0d50 con 0x7f11040b4d60 2024-03-25T19:33:27.111 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637522912 0 0) 0x7f11040b0d50 con 0x7f11040b4d60 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 >> v1:172.21.15.134:6790/0 conn(0x7f11040b1730 legacy=0x7f11040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 >> v1:172.21.15.134:6789/0 conn(0x7f110414be90 legacy=0x7f110414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 10 monclient: _finish_hunting 0 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 1 monclient: found mon.noname-c 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11040b1030 con 0x7f11040b4d60 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 10 monclient: _finish_auth 0 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 10 monclient: _check_auth_tickets 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f11000015b0 con 0x7f11040b4d60 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:27.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f1108eef640 10 monclient: dump: 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:27.113 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:27.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f110c155640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:33:27.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.111+0000 7f110c155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:27.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1100001710 con 0x7f11040b4d60 2024-03-25T19:33:27.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f1108eef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:27.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f1108eef640 1 -- 172.21.15.134:0/1000542165 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1100001b60 con 0x7f11040b4d60 2024-03-25T19:33:27.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f110c155640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:27.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f110c155640 4 set_mon_vals no callback set 2024-03-25T19:33:27.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f1108eef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f1108eef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.131+0000 7f1108eef640 10 monclient: dump: 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:27.132 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:27.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.132+0000 7f110c155640 10 monclient: shutdown 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.132+0000 7f110c155640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.132+0000 7f110c155640 1 -- 172.21.15.134:0/1000542165 >> v1:172.21.15.159:6789/0 conn(0x7f11040b4d60 legacy=0x7f110413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.132+0000 7f110c155640 1 -- 172.21.15.134:0/1000542165 shutdown_connections 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.132+0000 7f110c155640 1 -- 172.21.15.134:0/1000542165 >> 172.21.15.134:0/1000542165 conn(0x7f11040617a0 msgr2=0x7f1104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.132+0000 7f110c155640 1 -- 172.21.15.134:0/1000542165 shutdown_connections 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.132+0000 7f110c155640 1 -- 172.21.15.134:0/1000542165 wait complete. 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 10 monclient: build_initial_monmap 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 10 monclient: monmap: 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:27.135 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 1 librados: starting msgr at 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 1 librados: starting objecter 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 1 Processor -- start 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 1 -- start start 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 1 librados: setting wanted keys 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 1 librados: calling monclient init 2024-03-25T19:33:27.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.133+0000 7f110c155640 10 monclient: init 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 10 monclient(hunting): picked mon.noname-b con 0x7f11040b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 10 monclient(hunting): picked mon.noname-a con 0x7f11040b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 10 monclient(hunting): picked mon.noname-c con 0x7f110414be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11041d84a0 con 0x7f11040b4d60 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11041d7870 con 0x7f11040b1730 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110413b000 con 0x7f110414be90 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:27.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.134+0000 7f110c155640 10 monclient(hunting): authenticate will time out at 6239.293140s 2024-03-25T19:33:27.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.146+0000 7f110a6f2640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f110414be90 0x7f1104134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50826/0 (socket says 172.21.15.134:50826) 2024-03-25T19:33:27.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.146+0000 7f110a6f2640 1 -- 172.21.15.134:0/3576212776 learned_addr learned my addr 172.21.15.134:0/3576212776 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:27.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.147+0000 7f10fa7fc640 10 client.?.objecter ms_handle_connect 0x7f110414be90 2024-03-25T19:33:27.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.147+0000 7f10fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:27.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.152+0000 7f10fa7fc640 10 client.?.objecter ms_handle_connect 0x7f11040b1730 2024-03-25T19:33:27.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.152+0000 7f10fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:27.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.158+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830230895 0 0) 0x7f110413b000 con 0x7f110414be90 2024-03-25T19:33:27.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.158+0000 7f10fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:27.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.158+0000 7f10fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:27.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.158+0000 7f10fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:27.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.158+0000 7f10fa7fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:33:27.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.158+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e8004eb0 con 0x7f110414be90 2024-03-25T19:33:27.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.159+0000 7f10fa7fc640 10 client.?.objecter ms_handle_connect 0x7f11040b4d60 2024-03-25T19:33:27.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.159+0000 7f10fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:27.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.162+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709145738 0 0) 0x7f11041d7870 con 0x7f11040b1730 2024-03-25T19:33:27.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.162+0000 7f10fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:27.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.162+0000 7f10fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:27.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.162+0000 7f10fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:27.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.162+0000 7f10fa7fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:33:27.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.162+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110413b000 con 0x7f11040b1730 2024-03-25T19:33:27.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.174+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1413318929 0 0) 0x7f110413b000 con 0x7f11040b1730 2024-03-25T19:33:27.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.174+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11041d7870 con 0x7f11040b1730 2024-03-25T19:33:27.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.174+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1100001a20 con 0x7f11040b1730 2024-03-25T19:33:27.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.174+0000 7f10fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:27.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.174+0000 7f10fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:27.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.174+0000 7f10fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:27.174 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:27.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.174+0000 7f10fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:27.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.178+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880315907 0 0) 0x7f11041d84a0 con 0x7f11040b4d60 2024-03-25T19:33:27.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.178+0000 7f10fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:27.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.178+0000 7f10fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:27.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.178+0000 7f10fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:27.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.178+0000 7f10fa7fc640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:33:27.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.178+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110413b000 con 0x7f11040b4d60 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.180+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2743085879 0 0) 0x7f10e8004eb0 con 0x7f110414be90 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.180+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11041d84a0 con 0x7f110414be90 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.180+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f10fc001b70 con 0x7f110414be90 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.180+0000 7f10fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.180+0000 7f10fa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.180+0000 7f10fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:27.180 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:27.181 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:27.181 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:27.181 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:27.181 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:27.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2246301583 0 0) 0x7f11041d7870 con 0x7f11040b1730 2024-03-25T19:33:27.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 >> v1:172.21.15.134:6790/0 conn(0x7f110414be90 legacy=0x7f1104134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:27.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 >> v1:172.21.15.134:6789/0 conn(0x7f11040b4d60 legacy=0x7f11041d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:27.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:27.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 1 monclient: found mon.b 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11041355c0 con 0x7f11040b1730 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f10fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 5 monclient: authenticate success, global_id 4429 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 10 monclient: _renew_subs 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1104135810 con 0x7f11040b1730 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 10 monclient: _renew_subs 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f110413bc60 con 0x7f11040b1730 2024-03-25T19:33:27.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.181+0000 7f110c155640 1 librados: init done 2024-03-25T19:33:27.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.182+0000 7f110c155640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:27.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.182+0000 7f110c155640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:27.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.182+0000 7f110c155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:27.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.182+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11041d76b0 con 0x7f11040b1730 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.455+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1100001be0 con 0x7f11040b1730 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.455+0000 7f10fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.455+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1100003a40 con 0x7f11040b1730 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.455+0000 7f10fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.455+0000 7f10fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.455+0000 7f10fa7fc640 10 monclient: dump: 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.456 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:27.457 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:27.457 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:27.457 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:27.457 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:27.457 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:27.457 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:27.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.455+0000 7f10fbfff640 4 set_mon_vals no callback set 2024-03-25T19:33:27.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.456+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f1100019a10 con 0x7f11040b1730 2024-03-25T19:33:27.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.457+0000 7f11096f0640 10 monclient: get_auth_request con 0x7f10e806dfb0 auth_method 0 2024-03-25T19:33:27.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.457+0000 7f10fa7fc640 10 client.4429.objecter ms_handle_connect 0x7f10e806dfb0 2024-03-25T19:33:27.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.461+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f11000536a0 con 0x7f11040b1730 2024-03-25T19:33:27.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.461+0000 7f10fa7fc640 10 client.4429.objecter ms_dispatch 0x7f1104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:27.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.461+0000 7f10fa7fc640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:27.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.461+0000 7f10fa7fc640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:27.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.461+0000 7f10fa7fc640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T19:33:27.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.462+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f110004efe0 con 0x7f11040b1730 2024-03-25T19:33:27.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.462+0000 7f10fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:27.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.462+0000 7f10fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:27.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.694+0000 7f110c155640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:27.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.694+0000 7f110c155640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:27.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.694+0000 7f110c155640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:27.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.694+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11041d7a00 con 0x7f11040b1730 2024-03-25T19:33:27.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.724+0000 7f10fa7fc640 1 -- 172.21.15.134:0/3576212776 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f11000510e0 con 0x7f11040b1730 2024-03-25T19:33:27.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.724+0000 7f10fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:27.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.724+0000 7f10fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:27.725 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:27.725 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 10 librados: watch_flush enter 2024-03-25T19:33:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 10 librados: watch_flush exit 2024-03-25T19:33:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T19:33:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 10 client.4429.objecter successfully canceled tick 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 >> v1:172.21.15.159:6800/1875874887 conn(0x7f10e806dfb0 legacy=0x7f10e8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 10 monclient: shutdown 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 >> v1:172.21.15.159:6789/0 conn(0x7f11040b1730 legacy=0x7f1104140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 shutdown_connections 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 >> 172.21.15.134:0/3576212776 conn(0x7f11040617a0 msgr2=0x7f11040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 shutdown_connections 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 1 -- 172.21.15.134:0/3576212776 wait complete. 2024-03-25T19:33:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:27.727+0000 7f110c155640 1 librados: shutdown 2024-03-25T19:33:27.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:34.546 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:34.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.650+0000 7f4f5377a640 10 monclient: get_monmap_and_config 2024-03-25T19:33:34.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.650+0000 7f4f5377a640 10 monclient: build_initial_monmap 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.650+0000 7f4f5377a640 10 monclient: monmap: 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:34.651815+0000 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:34.651815+0000 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:34.651 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:34.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.651+0000 7f4f5377a640 1 Processor -- start 2024-03-25T19:33:34.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.651+0000 7f4f5377a640 1 -- start start 2024-03-25T19:33:34.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.651+0000 7f4f5377a640 10 monclient: init 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 10 monclient(hunting): picked mon.noname-c con 0x7f4f4c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4c0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 10 monclient(hunting): picked mon.noname-a con 0x7f4f4c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0af940 con 0x7f4f4c14be90 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0b0a70 con 0x7f4f4c0b4d60 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0b0d50 con 0x7f4f4c0b1730 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:34.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.652+0000 7f4f5377a640 10 monclient(hunting): authenticate will time out at 6246.810766s 2024-03-25T19:33:34.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.663+0000 7f4f51516640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f4f4c0b4d60 0x7f4f4c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:59902/0 (socket says 172.21.15.134:59902) 2024-03-25T19:33:34.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.663+0000 7f4f51516640 1 -- 172.21.15.134:0/1722100127 learned_addr learned my addr 172.21.15.134:0/1722100127 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:34.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.682+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328968340 0 0) 0x7f4f4c0b0d50 con 0x7f4f4c0b1730 2024-03-25T19:33:34.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.682+0000 7f4f43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:34.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.682+0000 7f4f43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:34.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.682+0000 7f4f43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:34.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.682+0000 7f4f43fff640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:33:34.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.682+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f38003520 con 0x7f4f4c0b1730 2024-03-25T19:33:34.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3568471252 0 0) 0x7f4f38003520 con 0x7f4f4c0b1730 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 >> v1:172.21.15.159:6789/0 conn(0x7f4f4c0b4d60 legacy=0x7f4f4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 >> v1:172.21.15.134:6789/0 conn(0x7f4f4c14be90 legacy=0x7f4f4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 1 monclient: found mon.noname-b 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0b1030 con 0x7f4f4c0b1730 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 10 monclient: _finish_auth 0 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:34.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4f3c001690 con 0x7f4f4c0b1730 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f43fff640 10 monclient: dump: 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f5377a640 5 monclient: authenticate success, global_id 4439 2024-03-25T19:33:34.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.688+0000 7f4f5377a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f3c0017f0 con 0x7f4f4c0b1730 2024-03-25T19:33:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f43fff640 1 -- 172.21.15.134:0/1722100127 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4f3c001c00 con 0x7f4f4c0b1730 2024-03-25T19:33:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f43fff640 10 monclient: dump: 2024-03-25T19:33:34.710 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f5377a640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:34.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.710+0000 7f4f5377a640 4 set_mon_vals no callback set 2024-03-25T19:33:34.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.711+0000 7f4f5377a640 10 monclient: shutdown 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.711+0000 7f4f5377a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.711+0000 7f4f5377a640 1 -- 172.21.15.134:0/1722100127 >> v1:172.21.15.134:6790/0 conn(0x7f4f4c0b1730 legacy=0x7f4f4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.711+0000 7f4f5377a640 1 -- 172.21.15.134:0/1722100127 shutdown_connections 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.711+0000 7f4f5377a640 1 -- 172.21.15.134:0/1722100127 >> 172.21.15.134:0/1722100127 conn(0x7f4f4c0617a0 msgr2=0x7f4f4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.711+0000 7f4f5377a640 1 -- 172.21.15.134:0/1722100127 shutdown_connections 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.711+0000 7f4f5377a640 1 -- 172.21.15.134:0/1722100127 wait complete. 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.712+0000 7f4f5377a640 10 monclient: build_initial_monmap 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.712+0000 7f4f5377a640 10 monclient: monmap: 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:34.713 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.712+0000 7f4f5377a640 1 librados: starting msgr at 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.712+0000 7f4f5377a640 1 librados: starting objecter 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 1 Processor -- start 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 1 -- start start 2024-03-25T19:33:34.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 1 librados: setting wanted keys 2024-03-25T19:33:34.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 1 librados: calling monclient init 2024-03-25T19:33:34.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient: init 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient(hunting): picked mon.noname-a con 0x7f4f4c0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient(hunting): picked mon.noname-c con 0x7f4f4c14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c137d90 con 0x7f4f4c0b4d60 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c142110 con 0x7f4f4c0b1730 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c140b20 con 0x7f4f4c14be90 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:34.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.713+0000 7f4f5377a640 10 monclient(hunting): authenticate will time out at 6246.872428s 2024-03-25T19:33:34.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.719+0000 7f4f51d17640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f4f4c14be90 0x7f4f4c1e3540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:40698/0 (socket says 172.21.15.134:40698) 2024-03-25T19:33:34.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.719+0000 7f4f51d17640 1 -- 172.21.15.134:0/53054410 learned_addr learned my addr 172.21.15.134:0/53054410 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:34.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.720+0000 7f4f41ffb640 10 client.?.objecter ms_handle_connect 0x7f4f4c14be90 2024-03-25T19:33:34.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.720+0000 7f4f41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:34.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.728+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615892866 0 0) 0x7f4f4c140b20 con 0x7f4f4c14be90 2024-03-25T19:33:34.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.728+0000 7f4f41ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:34.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.728+0000 7f4f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:34.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.728+0000 7f4f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:34.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.728+0000 7f4f41ffb640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:33:34.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.728+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f30004eb0 con 0x7f4f4c14be90 2024-03-25T19:33:34.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.730+0000 7f4f41ffb640 10 client.?.objecter ms_handle_connect 0x7f4f4c0b4d60 2024-03-25T19:33:34.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.730+0000 7f4f41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:34.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.737+0000 7f4f41ffb640 10 client.?.objecter ms_handle_connect 0x7f4f4c0b1730 2024-03-25T19:33:34.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.737+0000 7f4f41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:34.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.747+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545531878 0 0) 0x7f4f30004eb0 con 0x7f4f4c14be90 2024-03-25T19:33:34.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.747+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f4c140b20 con 0x7f4f4c14be90 2024-03-25T19:33:34.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.747+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4f44001c90 con 0x7f4f4c14be90 2024-03-25T19:33:34.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.747+0000 7f4f41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:34.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.747+0000 7f4f41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:34.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.747+0000 7f4f41ffb640 10 monclient(hunting): dump: 2024-03-25T19:33:34.747 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:34.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.747+0000 7f4f41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.752+0000 7f4f51516640 1 -- 172.21.15.134:0/53054410 >> v1:172.21.15.159:6789/0 conn(0x7f4f4c0b1730 legacy=0x7f4f4c140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.752+0000 7f4f51516640 1 -- 172.21.15.134:0/53054410 >> v1:172.21.15.159:6789/0 conn(0x7f4f4c0b1730 legacy=0x7f4f4c140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.752+0000 7f4f51516640 1 --1- 172.21.15.134:0/53054410 >> v1:172.21.15.159:6789/0 conn(0x7f4f4c0b1730 0x7f4f4c140410 :-1 s=OPENED pgs=35930 cs=1 l=1).handle_message read tag failed 2024-03-25T19:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.752+0000 7f4f51516640 1 --1- 172.21.15.134:0/53054410 >> v1:172.21.15.159:6789/0 conn(0x7f4f4c0b1730 0x7f4f4c140410 :-1 s=OPENED pgs=35930 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.752+0000 7f4f41ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.159:6789/0 2024-03-25T19:33:34.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.758+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982228986 0 0) 0x7f4f4c137d90 con 0x7f4f4c0b4d60 2024-03-25T19:33:34.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.758+0000 7f4f41ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:34.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.758+0000 7f4f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:34.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.758+0000 7f4f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:34.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.758+0000 7f4f41ffb640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:33:34.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.759+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f30004eb0 con 0x7f4f4c0b4d60 2024-03-25T19:33:34.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2468847240 0 0) 0x7f4f4c140b20 con 0x7f4f4c14be90 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 >> v1:172.21.15.159:6789/0 conn(0x7f4f4c0b1730 legacy=0x7f4f4c140410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 >> v1:172.21.15.134:6789/0 conn(0x7f4f4c0b4d60 legacy=0x7f4f4c133fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 1 monclient: found mon.c 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c140cb0 con 0x7f4f4c14be90 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 5 monclient: authenticate success, global_id 4442 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 10 monclient: _renew_subs 2024-03-25T19:33:34.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c140ef0 con 0x7f4f4c14be90 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 10 monclient: _renew_subs 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c142710 con 0x7f4f4c14be90 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.775+0000 7f4f5377a640 1 librados: init done 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.777+0000 7f4f5377a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.777+0000 7f4f5377a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:34.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.777+0000 7f4f5377a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:34.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:34.777+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c0b2270 con 0x7f4f4c14be90 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.087+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f44003990 con 0x7f4f4c14be90 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.087+0000 7f4f41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.087+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4f44003d20 con 0x7f4f4c14be90 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.087+0000 7f4f41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.087+0000 7f4f41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.087+0000 7f4f41ffb640 10 monclient: dump: 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:35.087 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:35.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.087+0000 7f4f437fe640 4 set_mon_vals no callback set 2024-03-25T19:33:35.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.088+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f4f44003f40 con 0x7f4f4c14be90 2024-03-25T19:33:35.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.088+0000 7f4f51516640 10 monclient: get_auth_request con 0x7f4f3006dd30 auth_method 0 2024-03-25T19:33:35.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.088+0000 7f4f41ffb640 10 client.4442.objecter ms_handle_connect 0x7f4f3006dd30 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f4f44054a30 con 0x7f4f4c14be90 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 10 client.4442.objecter ms_dispatch 0x7f4f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f44050540 con 0x7f4f4c14be90 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:35.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.094+0000 7f4f41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:35.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.342+0000 7f4f5377a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:35.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.342+0000 7f4f5377a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:35.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.342+0000 7f4f5377a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:35.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.342+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f4c0627b0 con 0x7f4f4c14be90 2024-03-25T19:33:35.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.375+0000 7f4f41ffb640 1 -- 172.21.15.134:0/53054410 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f4f44052470 con 0x7f4f4c14be90 2024-03-25T19:33:35.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.375+0000 7f4f41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:35.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.375+0000 7f4f41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:35.376 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:35.376 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:35.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 10 librados: watch_flush enter 2024-03-25T19:33:35.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 10 librados: watch_flush exit 2024-03-25T19:33:35.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T19:33:35.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 10 client.4442.objecter successfully canceled tick 2024-03-25T19:33:35.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 >> v1:172.21.15.159:6800/1875874887 conn(0x7f4f3006dd30 legacy=0x7f4f300701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 10 monclient: shutdown 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 >> v1:172.21.15.134:6790/0 conn(0x7f4f4c14be90 legacy=0x7f4f4c1e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 shutdown_connections 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 >> 172.21.15.134:0/53054410 conn(0x7f4f4c0617a0 msgr2=0x7f4f4c13b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 shutdown_connections 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 1 -- 172.21.15.134:0/53054410 wait complete. 2024-03-25T19:33:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:35.378+0000 7f4f5377a640 1 librados: shutdown 2024-03-25T19:33:35.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:42.198 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:42.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.317+0000 7f06737f6640 10 monclient: get_monmap_and_config 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.317+0000 7f06737f6640 10 monclient: build_initial_monmap 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.317+0000 7f06737f6640 10 monclient: monmap: 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:42.318010+0000 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:42.318010+0000 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:42.317 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:42.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.318+0000 7f06737f6640 1 Processor -- start 2024-03-25T19:33:42.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.318+0000 7f06737f6640 1 -- start start 2024-03-25T19:33:42.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.318+0000 7f06737f6640 10 monclient: init 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.319+0000 7f06737f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.319+0000 7f06737f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.319+0000 7f06737f6640 10 monclient(hunting): picked mon.noname-b con 0x7f066c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.319+0000 7f06737f6640 10 monclient(hunting): picked mon.noname-c con 0x7f066c13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.319+0000 7f06737f6640 10 monclient(hunting): picked mon.noname-a con 0x7f066c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.320+0000 7f06737f6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c142520 con 0x7f066c1430c0 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.320+0000 7f06737f6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c0b28a0 con 0x7f066c13f710 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.320+0000 7f06737f6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c0b2a30 con 0x7f066c0a98f0 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.320+0000 7f06737f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:42.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.320+0000 7f06737f6640 10 monclient(hunting): authenticate will time out at 6254.477730s 2024-03-25T19:33:42.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.330+0000 7f0670d91640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f066c13f710 0x7f066c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:50788/0 (socket says 172.21.15.134:50788) 2024-03-25T19:33:42.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.330+0000 7f0670d91640 1 -- 172.21.15.134:0/3323434016 learned_addr learned my addr 172.21.15.134:0/3323434016 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37938226 0 0) 0x7f066c0b28a0 con 0x7f066c13f710 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0658003520 con 0x7f066c13f710 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798957203 0 0) 0x7f066c142520 con 0x7f066c1430c0 2024-03-25T19:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.342+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c0b28a0 con 0x7f066c1430c0 2024-03-25T19:33:42.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.352+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703196073 0 0) 0x7f066c0b2a30 con 0x7f066c0a98f0 2024-03-25T19:33:42.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.352+0000 7f0663fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:42.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.352+0000 7f0663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:42.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.352+0000 7f0663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:42.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.352+0000 7f0663fff640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:33:42.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.352+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c142520 con 0x7f066c0a98f0 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1859116261 0 0) 0x7f066c0b28a0 con 0x7f066c1430c0 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 >> v1:172.21.15.134:6790/0 conn(0x7f066c0a98f0 legacy=0x7f066c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 >> v1:172.21.15.159:6789/0 conn(0x7f066c13f710 legacy=0x7f066c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 1 monclient: found mon.noname-a 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c0b44c0 con 0x7f066c1430c0 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 10 monclient: _finish_auth 0 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f06640019f0 con 0x7f066c1430c0 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f06737f6640 5 monclient: authenticate success, global_id 4479 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f06737f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:42.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f0663fff640 10 monclient: dump: 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:42.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.362+0000 7f06737f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0664001b50 con 0x7f066c1430c0 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f0663fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f0663fff640 1 -- 172.21.15.134:0/3323434016 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0664001f60 con 0x7f066c1430c0 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f06737f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f06737f6640 4 set_mon_vals no callback set 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f0663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f0663fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f0663fff640 10 monclient: dump: 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:42.373 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:42.374 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:42.374 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:42.374 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f06737f6640 10 monclient: shutdown 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f06737f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.374+0000 7f06737f6640 1 -- 172.21.15.134:0/3323434016 >> v1:172.21.15.134:6789/0 conn(0x7f066c1430c0 legacy=0x7f066c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.375+0000 7f06737f6640 1 -- 172.21.15.134:0/3323434016 shutdown_connections 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.375+0000 7f06737f6640 1 -- 172.21.15.134:0/3323434016 >> 172.21.15.134:0/3323434016 conn(0x7f066c0b7ea0 msgr2=0x7f066c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.375+0000 7f06737f6640 1 -- 172.21.15.134:0/3323434016 shutdown_connections 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.375+0000 7f06737f6640 1 -- 172.21.15.134:0/3323434016 wait complete. 2024-03-25T19:33:42.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 10 monclient: build_initial_monmap 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 10 monclient: monmap: 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 1 librados: starting msgr at 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 1 librados: starting objecter 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 1 Processor -- start 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 1 -- start start 2024-03-25T19:33:42.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 1 librados: setting wanted keys 2024-03-25T19:33:42.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 1 librados: calling monclient init 2024-03-25T19:33:42.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.376+0000 7f06737f6640 10 monclient: init 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 10 monclient(hunting): picked mon.noname-c con 0x7f066c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 10 monclient(hunting): picked mon.noname-b con 0x7f066c13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 10 monclient(hunting): picked mon.noname-a con 0x7f066c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c138bd0 con 0x7f066c1430c0 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c0aba30 con 0x7f066c13f710 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c137740 con 0x7f066c0a98f0 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:42.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.377+0000 7f06737f6640 10 monclient(hunting): authenticate will time out at 6254.535305s 2024-03-25T19:33:42.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.384+0000 7f0671d93640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f066c1430c0 0x7f066c1d7780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:59264/0 (socket says 172.21.15.134:59264) 2024-03-25T19:33:42.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.384+0000 7f0671d93640 1 -- 172.21.15.134:0/2500478335 learned_addr learned my addr 172.21.15.134:0/2500478335 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:42.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.385+0000 7f0661ffb640 10 client.?.objecter ms_handle_connect 0x7f066c1430c0 2024-03-25T19:33:42.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.385+0000 7f0661ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:42.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.393+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519081558 0 0) 0x7f066c138bd0 con 0x7f066c1430c0 2024-03-25T19:33:42.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.393+0000 7f0661ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:42.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.393+0000 7f0661ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:42.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.393+0000 7f0661ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:42.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.393+0000 7f0661ffb640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:33:42.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.393+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0650004eb0 con 0x7f066c1430c0 2024-03-25T19:33:42.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.398+0000 7f0661ffb640 10 client.?.objecter ms_handle_connect 0x7f066c0a98f0 2024-03-25T19:33:42.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.398+0000 7f0661ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:42.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.403+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433709832 0 0) 0x7f0650004eb0 con 0x7f066c1430c0 2024-03-25T19:33:42.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.403+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f066c138bd0 con 0x7f066c1430c0 2024-03-25T19:33:42.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.407+0000 7f0661ffb640 10 client.?.objecter ms_handle_connect 0x7f066c13f710 2024-03-25T19:33:42.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.407+0000 7f0661ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:42.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.413+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689393014 0 0) 0x7f066c137740 con 0x7f066c0a98f0 2024-03-25T19:33:42.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.413+0000 7f0661ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:42.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.413+0000 7f0661ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:42.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.413+0000 7f0661ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:42.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.413+0000 7f0661ffb640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:33:42.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.413+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0650004eb0 con 0x7f066c0a98f0 2024-03-25T19:33:42.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f06640020a0 con 0x7f066c1430c0 2024-03-25T19:33:42.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:42.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:42.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient(hunting): dump: 2024-03-25T19:33:42.414 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:42.414 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:42.414 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1374703735 0 0) 0x7f066c138bd0 con 0x7f066c1430c0 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 >> v1:172.21.15.134:6790/0 conn(0x7f066c0a98f0 legacy=0x7f066c1384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 >> v1:172.21.15.159:6789/0 conn(0x7f066c13f710 legacy=0x7f066c134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 1 monclient: found mon.a 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:33:42.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c134de0 con 0x7f066c1430c0 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f0661ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 5 monclient: authenticate success, global_id 4482 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 10 monclient: _renew_subs 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c0b1620 con 0x7f066c1430c0 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 10 monclient: _renew_subs 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f066c134900 con 0x7f066c1430c0 2024-03-25T19:33:42.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.415+0000 7f06737f6640 1 librados: init done 2024-03-25T19:33:42.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.416+0000 7f06737f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:42.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.416+0000 7f06737f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:42.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.416+0000 7f06737f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:33:42.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.416+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066c1d75d0 con 0x7f066c1430c0 2024-03-25T19:33:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.730+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0664003ca0 con 0x7f066c1430c0 2024-03-25T19:33:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.730+0000 7f0661ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.730+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0664004070 con 0x7f066c1430c0 2024-03-25T19:33:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.730+0000 7f0661ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.730+0000 7f0661ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.730+0000 7f0661ffb640 10 monclient: dump: 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:42.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.730+0000 7f06637fe640 4 set_mon_vals no callback set 2024-03-25T19:33:42.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.731+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f0664004290 con 0x7f066c1430c0 2024-03-25T19:33:42.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.731+0000 7f0671592640 10 monclient: get_auth_request con 0x7f065006ddd0 auth_method 0 2024-03-25T19:33:42.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.732+0000 7f0661ffb640 10 client.4482.objecter ms_handle_connect 0x7f065006ddd0 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.736+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f0664054c20 con 0x7f066c1430c0 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.736+0000 7f0661ffb640 10 client.4482.objecter ms_dispatch 0x7f066c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.736+0000 7f0661ffb640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.736+0000 7f0661ffb640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.737+0000 7f0661ffb640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.737+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0664050730 con 0x7f066c1430c0 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.737+0000 7f0661ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:42.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.737+0000 7f0661ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:42.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.954+0000 7f06737f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:42.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.954+0000 7f06737f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:42.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.954+0000 7f06737f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:33:42.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.954+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f066c0627b0 con 0x7f066c1430c0 2024-03-25T19:33:42.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.981+0000 7f0661ffb640 1 -- 172.21.15.134:0/2500478335 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f0664052660 con 0x7f066c1430c0 2024-03-25T19:33:42.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.981+0000 7f0661ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:42.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.981+0000 7f0661ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:42.981 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:42.981 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 10 librados: watch_flush enter 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 10 librados: watch_flush exit 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 10 client.4482.objecter successfully canceled tick 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 >> v1:172.21.15.159:6800/1875874887 conn(0x7f065006ddd0 legacy=0x7f0650070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 10 monclient: shutdown 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 >> v1:172.21.15.134:6789/0 conn(0x7f066c1430c0 legacy=0x7f066c1d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 shutdown_connections 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 >> 172.21.15.134:0/2500478335 conn(0x7f066c0b7ea0 msgr2=0x7f066c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.983+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 shutdown_connections 2024-03-25T19:33:42.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.984+0000 7f06737f6640 1 -- 172.21.15.134:0/2500478335 wait complete. 2024-03-25T19:33:42.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:42.984+0000 7f06737f6640 1 librados: shutdown 2024-03-25T19:33:42.996 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:45.418 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:50.090 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.438 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:50.110 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.473 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:50.145 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.480 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:50.152 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.485 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:50.157 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.490 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:50.137 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.494 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:50.141 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.516 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:50.163 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:45.520 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:50.193 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:49.802 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:49.913 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.914+0000 7f641b445640 10 monclient: get_monmap_and_config 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.914+0000 7f641b445640 10 monclient: build_initial_monmap 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.914+0000 7f641b445640 10 monclient: monmap: 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:49.915111+0000 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:49.915111+0000 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:49.914 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:49.915 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:49.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.914+0000 7f641b445640 1 Processor -- start 2024-03-25T19:33:49.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.914+0000 7f641b445640 1 -- start start 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.914+0000 7f641b445640 10 monclient: init 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 10 monclient(hunting): picked mon.noname-a con 0x7f64140a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 10 monclient(hunting): picked mon.noname-b con 0x7f641413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 10 monclient(hunting): picked mon.noname-c con 0x7f64141430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6414142520 con 0x7f64140a98f0 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64140b28a0 con 0x7f64141430c0 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64140b2a30 con 0x7f641413f710 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.915+0000 7f641b445640 10 monclient(hunting): authenticate will time out at 6262.074514s 2024-03-25T19:33:49.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.919+0000 7f64199e2640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f64141430c0 0x7f641414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54990/0 (socket says 172.21.15.134:54990) 2024-03-25T19:33:49.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.919+0000 7f64199e2640 1 -- 172.21.15.134:0/3676280828 learned_addr learned my addr 172.21.15.134:0/3676280828 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:49.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.927+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157693786 0 0) 0x7f64140b28a0 con 0x7f64141430c0 2024-03-25T19:33:49.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.927+0000 7f640bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.927+0000 7f640bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.927+0000 7f640bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.927+0000 7f640bfff640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:33:49.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.927+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6400003520 con 0x7f64141430c0 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4194200130 0 0) 0x7f6400003520 con 0x7f64141430c0 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 >> v1:172.21.15.134:6790/0 conn(0x7f641413f710 legacy=0x7f64141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 >> v1:172.21.15.134:6789/0 conn(0x7f64140a98f0 legacy=0x7f64140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 1 monclient: found mon.noname-c 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64140b44c0 con 0x7f64141430c0 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 10 monclient: _finish_auth 0 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f640c001a10 con 0x7f64141430c0 2024-03-25T19:33:49.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f640bfff640 10 monclient: dump: 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f641b445640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:33:49.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.936+0000 7f641b445640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f640c001b70 con 0x7f64141430c0 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f640bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f640bfff640 1 -- 172.21.15.134:0/3676280828 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f640c001f80 con 0x7f64141430c0 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f641b445640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f641b445640 4 set_mon_vals no callback set 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f640bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f640bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f640bfff640 10 monclient: dump: 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:49.942 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.943 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.943 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.943 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:49.943 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:49.943 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:49.943 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:49.943 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:49.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f641b445640 10 monclient: shutdown 2024-03-25T19:33:49.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f641b445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f641b445640 1 -- 172.21.15.134:0/3676280828 >> v1:172.21.15.159:6789/0 conn(0x7f64141430c0 legacy=0x7f641414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f641b445640 1 -- 172.21.15.134:0/3676280828 shutdown_connections 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.942+0000 7f641b445640 1 -- 172.21.15.134:0/3676280828 >> 172.21.15.134:0/3676280828 conn(0x7f64140b7ea0 msgr2=0x7f64140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.943+0000 7f641b445640 1 -- 172.21.15.134:0/3676280828 shutdown_connections 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.943+0000 7f641b445640 1 -- 172.21.15.134:0/3676280828 wait complete. 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.943+0000 7f641b445640 10 monclient: build_initial_monmap 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.943+0000 7f641b445640 10 monclient: monmap: 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:49.946 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.943+0000 7f641b445640 1 librados: starting msgr at 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 librados: starting objecter 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 Processor -- start 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 -- start start 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 librados: setting wanted keys 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 librados: calling monclient init 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient: init 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient(hunting): picked mon.noname-b con 0x7f64140a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:49.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient(hunting): picked mon.noname-c con 0x7f641413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:49.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient(hunting): picked mon.noname-a con 0x7f64141430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:49.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64141396e0 con 0x7f64141430c0 2024-03-25T19:33:49.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6414138530 con 0x7f64140a98f0 2024-03-25T19:33:49.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64141399c0 con 0x7f641413f710 2024-03-25T19:33:49.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:49.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.944+0000 7f641b445640 10 monclient(hunting): authenticate will time out at 6262.103557s 2024-03-25T19:33:49.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.949+0000 7f64189e0640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f641413f710 0x7f64141da850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:41570/0 (socket says 172.21.15.134:41570) 2024-03-25T19:33:49.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.949+0000 7f64189e0640 1 -- 172.21.15.134:0/400041206 learned_addr learned my addr 172.21.15.134:0/400041206 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:49.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.954+0000 7f6409ffb640 10 client.?.objecter ms_handle_connect 0x7f641413f710 2024-03-25T19:33:49.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.954+0000 7f6409ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:49.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.960+0000 7f6409ffb640 10 client.?.objecter ms_handle_connect 0x7f64141430c0 2024-03-25T19:33:49.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.960+0000 7f6409ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:49.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.962+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093855471 0 0) 0x7f64141399c0 con 0x7f641413f710 2024-03-25T19:33:49.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.962+0000 7f6409ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.962+0000 7f6409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.962+0000 7f6409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.962+0000 7f6409ffb640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:33:49.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.962+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f8004eb0 con 0x7f641413f710 2024-03-25T19:33:49.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.964+0000 7f6409ffb640 10 client.?.objecter ms_handle_connect 0x7f64140a98f0 2024-03-25T19:33:49.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.964+0000 7f6409ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:49.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.981+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464688490 0 0) 0x7f64141396e0 con 0x7f64141430c0 2024-03-25T19:33:49.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.981+0000 7f6409ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.981+0000 7f6409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.981+0000 7f6409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.981+0000 7f6409ffb640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:33:49.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.981+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64141399c0 con 0x7f64141430c0 2024-03-25T19:33:49.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.984+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 29126736 0 0) 0x7f63f8004eb0 con 0x7f641413f710 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.984+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64141396e0 con 0x7f641413f710 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.984+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f64040019d0 con 0x7f641413f710 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.984+0000 7f6409ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.984+0000 7f6409ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.984+0000 7f6409ffb640 10 monclient(hunting): dump: 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:49.985 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.984+0000 7f6409ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.985+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050331752 0 0) 0x7f6414138530 con 0x7f64140a98f0 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.985+0000 7f6409ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.985+0000 7f6409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.985+0000 7f6409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.985+0000 7f6409ffb640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:33:49.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.985+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f8004eb0 con 0x7f64140a98f0 2024-03-25T19:33:49.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.991+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3093773269 0 0) 0x7f64141399c0 con 0x7f64141430c0 2024-03-25T19:33:49.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.991+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6414138530 con 0x7f64141430c0 2024-03-25T19:33:49.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.991+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f640c001e80 con 0x7f64141430c0 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.991+0000 7f6409ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.991+0000 7f6409ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:49.991+0000 7f6409ffb640 10 monclient(hunting): dump: 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:49.992 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:50.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3760274034 0 0) 0x7f64141396e0 con 0x7f641413f710 2024-03-25T19:33:50.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 >> v1:172.21.15.159:6789/0 conn(0x7f64140a98f0 legacy=0x7f6414137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 >> v1:172.21.15.134:6789/0 conn(0x7f64141430c0 legacy=0x7f64141e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 1 monclient: found mon.c 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414138a50 con 0x7f641413f710 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f6409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 5 monclient: authenticate success, global_id 4454 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 10 monclient: _renew_subs 2024-03-25T19:33:50.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64141380a0 con 0x7f641413f710 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 10 monclient: _renew_subs 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64141db140 con 0x7f641413f710 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.003+0000 7f641b445640 1 librados: init done 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.004+0000 7f641b445640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.004+0000 7f641b445640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.004+0000 7f641b445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:50.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.004+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6414138ca0 con 0x7f641413f710 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6404003610 con 0x7f641413f710 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f6409ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f6404003800 con 0x7f641413f710 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f6409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f6409ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f6409ffb640 10 monclient: dump: 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:50.341 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f640b7fe640 4 set_mon_vals no callback set 2024-03-25T19:33:50.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.341+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f6404003a20 con 0x7f641413f710 2024-03-25T19:33:50.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.342+0000 7f64191e1640 10 monclient: get_auth_request con 0x7f63f806dfb0 auth_method 0 2024-03-25T19:33:50.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.342+0000 7f6409ffb640 10 client.4454.objecter ms_handle_connect 0x7f63f806dfb0 2024-03-25T19:33:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f6404054620 con 0x7f641413f710 2024-03-25T19:33:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 10 client.4454.objecter ms_dispatch 0x7f6414000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T19:33:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6404050130 con 0x7f641413f710 2024-03-25T19:33:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.346+0000 7f6409ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:50.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.591+0000 7f641b445640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:50.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.591+0000 7f641b445640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:50.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.591+0000 7f641b445640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:33:50.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.591+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64140627b0 con 0x7f641413f710 2024-03-25T19:33:50.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.624+0000 7f6409ffb640 1 -- 172.21.15.134:0/400041206 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f6404052060 con 0x7f641413f710 2024-03-25T19:33:50.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.624+0000 7f6409ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:50.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.624+0000 7f6409ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:50.624 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:50.624 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:50.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.626+0000 7f641b445640 10 librados: watch_flush enter 2024-03-25T19:33:50.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 10 librados: watch_flush exit 2024-03-25T19:33:50.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T19:33:50.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 10 client.4454.objecter successfully canceled tick 2024-03-25T19:33:50.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 >> v1:172.21.15.159:6800/1875874887 conn(0x7f63f806dfb0 legacy=0x7f63f8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 10 monclient: shutdown 2024-03-25T19:33:50.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:50.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 >> v1:172.21.15.134:6790/0 conn(0x7f641413f710 legacy=0x7f64141da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:50.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 shutdown_connections 2024-03-25T19:33:50.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 >> 172.21.15.134:0/400041206 conn(0x7f64140b7ea0 msgr2=0x7f6414074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:50.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 shutdown_connections 2024-03-25T19:33:50.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 1 -- 172.21.15.134:0/400041206 wait complete. 2024-03-25T19:33:50.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:50.627+0000 7f641b445640 1 librados: shutdown 2024-03-25T19:33:50.640 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:57.446 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:57.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.564+0000 7f4795a55640 10 monclient: get_monmap_and_config 2024-03-25T19:33:57.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.564+0000 7f4795a55640 10 monclient: build_initial_monmap 2024-03-25T19:33:57.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.564+0000 7f4795a55640 10 monclient: monmap: 2024-03-25T19:33:57.564 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:33:57.565523+0000 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:33:57.565523+0000 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:33:57.565 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.565+0000 7f4795a55640 1 Processor -- start 2024-03-25T19:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.565+0000 7f4795a55640 1 -- start start 2024-03-25T19:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.565+0000 7f4795a55640 10 monclient: init 2024-03-25T19:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 10 monclient(hunting): picked mon.noname-b con 0x7f47900b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 10 monclient(hunting): picked mon.noname-c con 0x7f47900b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 10 monclient(hunting): picked mon.noname-a con 0x7f479014be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900af940 con 0x7f479014be90 2024-03-25T19:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900b0a70 con 0x7f47900b1730 2024-03-25T19:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900b0d50 con 0x7f47900b4d60 2024-03-25T19:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.566+0000 7f4795a55640 10 monclient(hunting): authenticate will time out at 6269.725226s 2024-03-25T19:33:57.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.578+0000 7f478f7fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f479014be90 0x7f479014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:48292/0 (socket says 172.21.15.134:48292) 2024-03-25T19:33:57.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.578+0000 7f478f7fe640 1 -- 172.21.15.134:0/3320098294 learned_addr learned my addr 172.21.15.134:0/3320098294 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:57.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.597+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994067089 0 0) 0x7f47900af940 con 0x7f479014be90 2024-03-25T19:33:57.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.597+0000 7f478dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:57.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.597+0000 7f478dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:57.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.597+0000 7f478dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:57.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.597+0000 7f478dffb640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:33:57.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.597+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4778003520 con 0x7f479014be90 2024-03-25T19:33:57.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.598+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265419322 0 0) 0x7f47900b0d50 con 0x7f47900b4d60 2024-03-25T19:33:57.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.598+0000 7f478dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:57.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.598+0000 7f478dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:57.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.598+0000 7f478dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:57.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.598+0000 7f478dffb640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:33:57.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.598+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47900af940 con 0x7f47900b4d60 2024-03-25T19:33:57.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.601+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133778293 0 0) 0x7f47900b0a70 con 0x7f47900b1730 2024-03-25T19:33:57.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.601+0000 7f478dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:57.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.601+0000 7f478dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:57.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.601+0000 7f478dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:57.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.601+0000 7f478dffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:33:57.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.601+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47900b0d50 con 0x7f47900b1730 2024-03-25T19:33:57.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621012826 0 0) 0x7f47900af940 con 0x7f47900b4d60 2024-03-25T19:33:57.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 >> v1:172.21.15.159:6789/0 conn(0x7f47900b1730 legacy=0x7f47900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:57.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 >> v1:172.21.15.134:6789/0 conn(0x7f479014be90 legacy=0x7f479014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:57.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:57.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 1 monclient: found mon.noname-b 2024-03-25T19:33:57.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47900b1030 con 0x7f47900b4d60 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4780001670 con 0x7f47900b4d60 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f478dffb640 10 monclient: dump: 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:57.607 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:57.608 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:57.608 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:57.608 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:57.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f4795a55640 5 monclient: authenticate success, global_id 4457 2024-03-25T19:33:57.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.606+0000 7f4795a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47800017d0 con 0x7f47900b4d60 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f478dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f478dffb640 1 -- 172.21.15.134:0/3320098294 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4780001be0 con 0x7f47900b4d60 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 4 set_mon_vals no callback set 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f478dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f478dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f478dffb640 10 monclient: dump: 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.638 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.639 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:57.639 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:57.639 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:57.639 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:57.639 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:57.639 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:57.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 10 monclient: shutdown 2024-03-25T19:33:57.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:57.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 1 -- 172.21.15.134:0/3320098294 >> v1:172.21.15.134:6790/0 conn(0x7f47900b4d60 legacy=0x7f479013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:57.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 1 -- 172.21.15.134:0/3320098294 shutdown_connections 2024-03-25T19:33:57.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 1 -- 172.21.15.134:0/3320098294 >> 172.21.15.134:0/3320098294 conn(0x7f47900617a0 msgr2=0x7f4790061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.638+0000 7f4795a55640 1 -- 172.21.15.134:0/3320098294 shutdown_connections 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.639+0000 7f4795a55640 1 -- 172.21.15.134:0/3320098294 wait complete. 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.639+0000 7f4795a55640 10 monclient: build_initial_monmap 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.639+0000 7f4795a55640 10 monclient: monmap: 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:57.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.640+0000 7f4795a55640 1 librados: starting msgr at 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.640+0000 7f4795a55640 1 librados: starting objecter 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.640+0000 7f4795a55640 1 Processor -- start 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.640+0000 7f4795a55640 1 -- start start 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.640+0000 7f4795a55640 1 librados: setting wanted keys 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.640+0000 7f4795a55640 1 librados: calling monclient init 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.640+0000 7f4795a55640 10 monclient: init 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 10 monclient(hunting): picked mon.noname-a con 0x7f47900b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 10 monclient(hunting): picked mon.noname-c con 0x7f47900b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 10 monclient(hunting): picked mon.noname-b con 0x7f479014be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4790141e80 con 0x7f47900b1730 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47901d7570 con 0x7f479014be90 2024-03-25T19:33:57.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47901d81a0 con 0x7f47900b4d60 2024-03-25T19:33:57.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:57.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.641+0000 7f4795a55640 10 monclient(hunting): authenticate will time out at 6269.800162s 2024-03-25T19:33:57.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.651+0000 7f478f7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f479014be90 0x7f4790142a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58628/0 (socket says 172.21.15.134:58628) 2024-03-25T19:33:57.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.651+0000 7f478f7fe640 1 -- 172.21.15.134:0/1863614015 learned_addr learned my addr 172.21.15.134:0/1863614015 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:33:57.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.655+0000 7f4777fff640 10 client.?.objecter ms_handle_connect 0x7f47900b4d60 2024-03-25T19:33:57.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.655+0000 7f4777fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:57.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.657+0000 7f4777fff640 10 client.?.objecter ms_handle_connect 0x7f479014be90 2024-03-25T19:33:57.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.657+0000 7f4777fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:57.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.665+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692465092 0 0) 0x7f47901d7570 con 0x7f479014be90 2024-03-25T19:33:57.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.665+0000 7f4777fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:57.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.665+0000 7f4777fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:57.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.665+0000 7f4777fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:57.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.665+0000 7f4777fff640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:33:57.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.665+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f477c004eb0 con 0x7f479014be90 2024-03-25T19:33:57.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.666+0000 7f4777fff640 10 client.?.objecter ms_handle_connect 0x7f47900b1730 2024-03-25T19:33:57.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.666+0000 7f4777fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:57.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.672+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025909672 0 0) 0x7f47901d81a0 con 0x7f47900b4d60 2024-03-25T19:33:57.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.672+0000 7f4777fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:57.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.672+0000 7f4777fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:57.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.672+0000 7f4777fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:57.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.672+0000 7f4777fff640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:33:57.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.672+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47901d7570 con 0x7f47900b4d60 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.683+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1610728553 0 0) 0x7f477c004eb0 con 0x7f479014be90 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.683+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47901d81a0 con 0x7f479014be90 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.684+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4788001b30 con 0x7f479014be90 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.684+0000 7f4777fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.684+0000 7f4777fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.684+0000 7f4777fff640 10 monclient(hunting): dump: 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:57.684 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:57.685 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:57.685 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:57.685 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:57.685 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:57.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.684+0000 7f4777fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:57.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.691+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321612904 0 0) 0x7f4790141e80 con 0x7f47900b1730 2024-03-25T19:33:57.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.691+0000 7f4777fff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:57.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.691+0000 7f4777fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:57.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.691+0000 7f4777fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:57.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.691+0000 7f4777fff640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:33:57.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.691+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f477c004eb0 con 0x7f47900b1730 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1573364033 0 0) 0x7f47901d81a0 con 0x7f479014be90 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 >> v1:172.21.15.134:6790/0 conn(0x7f47900b4d60 legacy=0x7f47901d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 >> v1:172.21.15.134:6789/0 conn(0x7f47900b1730 legacy=0x7f4790138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 1 monclient: found mon.b 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47901d8330 con 0x7f479014be90 2024-03-25T19:33:57.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 10 monclient: _finish_auth 0 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4777fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 10 monclient: _renew_subs 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4790143110 con 0x7f479014be90 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 10 monclient: _renew_subs 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4790142480 con 0x7f479014be90 2024-03-25T19:33:57.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.695+0000 7f4795a55640 1 librados: init done 2024-03-25T19:33:57.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.696+0000 7f4795a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:57.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.697+0000 7f4795a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:57.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.697+0000 7f4795a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:57.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.697+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47901401f0 con 0x7f479014be90 2024-03-25T19:33:57.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.977+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4788001fe0 con 0x7f479014be90 2024-03-25T19:33:57.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.977+0000 7f4777fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:57.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.977+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4788004b60 con 0x7f479014be90 2024-03-25T19:33:57.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.977+0000 7f4777fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:57.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.977+0000 7f4777fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.977+0000 7f4777fff640 10 monclient: dump: 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:33:57.979 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:33:57.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.977+0000 7f478d7fa640 4 set_mon_vals no callback set 2024-03-25T19:33:57.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.978+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f4788004cc0 con 0x7f479014be90 2024-03-25T19:33:57.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.979+0000 7f478effd640 10 monclient: get_auth_request con 0x7f477c06de00 auth_method 0 2024-03-25T19:33:57.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.979+0000 7f4777fff640 10 client.4453.objecter ms_handle_connect 0x7f477c06de00 2024-03-25T19:33:57.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f4788054890 con 0x7f479014be90 2024-03-25T19:33:57.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 10 client.4453.objecter ms_dispatch 0x7f4790000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:57.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:57.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:57.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T19:33:57.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47880501d0 con 0x7f479014be90 2024-03-25T19:33:57.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:57.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:57.984+0000 7f4777fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:58.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.233+0000 7f4795a55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:58.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.233+0000 7f4795a55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:58.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.233+0000 7f4795a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:33:58.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.233+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4790141cc0 con 0x7f479014be90 2024-03-25T19:33:58.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.263+0000 7f4777fff640 1 -- 172.21.15.134:0/1863614015 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f47880522d0 con 0x7f479014be90 2024-03-25T19:33:58.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.263+0000 7f4777fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:58.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.263+0000 7f4777fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:58.264 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:33:58.264 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:58.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 10 librados: watch_flush enter 2024-03-25T19:33:58.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 10 librados: watch_flush exit 2024-03-25T19:33:58.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 10 client.4453.objecter successfully canceled tick 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 >> v1:172.21.15.159:6800/1875874887 conn(0x7f477c06de00 legacy=0x7f477c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 10 monclient: shutdown 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 >> v1:172.21.15.159:6789/0 conn(0x7f479014be90 legacy=0x7f4790142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 shutdown_connections 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 >> 172.21.15.134:0/1863614015 conn(0x7f47900617a0 msgr2=0x7f47900b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 shutdown_connections 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 1 -- 172.21.15.134:0/1863614015 wait complete. 2024-03-25T19:33:58.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:33:58.266+0000 7f4795a55640 1 librados: shutdown 2024-03-25T19:33:58.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:05.085 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:05.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.200+0000 7fc30c1d6640 10 monclient: get_monmap_and_config 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.200+0000 7fc30c1d6640 10 monclient: build_initial_monmap 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.200+0000 7fc30c1d6640 10 monclient: monmap: 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:05.201946+0000 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:05.201946+0000 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:05.201 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.201+0000 7fc30c1d6640 1 Processor -- start 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.201+0000 7fc30c1d6640 1 -- start start 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.201+0000 7fc30c1d6640 10 monclient: init 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3040a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc30413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc3041430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc304142520 con 0x7fc3041430c0 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3040b28a0 con 0x7fc30413f710 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3040b2a30 con 0x7fc3040a98f0 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:05.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.202+0000 7fc30c1d6640 10 monclient(hunting): authenticate will time out at 6277.361508s 2024-03-25T19:34:05.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.211+0000 7fc309f72640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc3040a98f0 0x7fc3040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:35792/0 (socket says 172.21.15.134:35792) 2024-03-25T19:34:05.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.211+0000 7fc309f72640 1 -- 172.21.15.134:0/643652796 learned_addr learned my addr 172.21.15.134:0/643652796 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:05.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.223+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041053873 0 0) 0x7fc3040b28a0 con 0x7fc30413f710 2024-03-25T19:34:05.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.223+0000 7fc308f70640 10 monclient(hunting): _negotiate 2024-03-25T19:34:05.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.223+0000 7fc308f70640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:05.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.223+0000 7fc308f70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:05.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.223+0000 7fc308f70640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:34:05.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.223+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0003520 con 0x7fc30413f710 2024-03-25T19:34:05.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.229+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528116348 0 0) 0x7fc3040b2a30 con 0x7fc3040a98f0 2024-03-25T19:34:05.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.229+0000 7fc308f70640 10 monclient(hunting): _negotiate 2024-03-25T19:34:05.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.229+0000 7fc308f70640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:05.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.229+0000 7fc308f70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:05.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.229+0000 7fc308f70640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:34:05.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.229+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3040b28a0 con 0x7fc3040a98f0 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268343826 0 0) 0x7fc2f0003520 con 0x7fc30413f710 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 >> v1:172.21.15.134:6790/0 conn(0x7fc3040a98f0 legacy=0x7fc3040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 >> v1:172.21.15.134:6789/0 conn(0x7fc3041430c0 legacy=0x7fc30414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 10 monclient: _finish_hunting 0 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 1 monclient: found mon.noname-c 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:34:05.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3040b44c0 con 0x7fc30413f710 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 10 monclient: _finish_auth 0 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 10 monclient: _check_auth_tickets 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc2f4001690 con 0x7fc30413f710 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc308f70640 10 monclient: dump: 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:05.235 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc30c1d6640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:34:05.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.234+0000 7fc30c1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:05.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2f40017f0 con 0x7fc30413f710 2024-03-25T19:34:05.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc308f70640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:05.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc308f70640 1 -- 172.21.15.134:0/643652796 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc2f4001c00 con 0x7fc30413f710 2024-03-25T19:34:05.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc308f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:05.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc308f70640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc308f70640 10 monclient: dump: 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc30c1d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:05.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.247+0000 7fc30c1d6640 4 set_mon_vals no callback set 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.248+0000 7fc30c1d6640 10 monclient: shutdown 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.248+0000 7fc30c1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.248+0000 7fc30c1d6640 1 -- 172.21.15.134:0/643652796 >> v1:172.21.15.159:6789/0 conn(0x7fc30413f710 legacy=0x7fc3041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.248+0000 7fc30c1d6640 1 -- 172.21.15.134:0/643652796 shutdown_connections 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.248+0000 7fc30c1d6640 1 -- 172.21.15.134:0/643652796 >> 172.21.15.134:0/643652796 conn(0x7fc3040b7ea0 msgr2=0x7fc3040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.248+0000 7fc30c1d6640 1 -- 172.21.15.134:0/643652796 shutdown_connections 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.248+0000 7fc30c1d6640 1 -- 172.21.15.134:0/643652796 wait complete. 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.249+0000 7fc30c1d6640 10 monclient: build_initial_monmap 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.249+0000 7fc30c1d6640 10 monclient: monmap: 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:05.251 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 librados: starting msgr at 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 librados: starting objecter 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 Processor -- start 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 -- start start 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 librados: setting wanted keys 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 librados: calling monclient init 2024-03-25T19:34:05.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 10 monclient: init 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3040a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc30413f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc3041430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc304135730 con 0x7fc30413f710 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.250+0000 7fc30c1d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3041358c0 con 0x7fc3040a98f0 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.251+0000 7fc30c1d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc304135a50 con 0x7fc3041430c0 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.251+0000 7fc30c1d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:05.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.251+0000 7fc30c1d6640 10 monclient(hunting): authenticate will time out at 6277.409660s 2024-03-25T19:34:05.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.259+0000 7fc309771640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc30413f710 0x7fc3041d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:42710/0 (socket says 172.21.15.134:42710) 2024-03-25T19:34:05.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.259+0000 7fc309771640 1 -- 172.21.15.134:0/4228898169 learned_addr learned my addr 172.21.15.134:0/4228898169 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:05.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.265+0000 7fc2fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc30413f710 2024-03-25T19:34:05.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.265+0000 7fc2fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:05.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.269+0000 7fc2fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc3040a98f0 2024-03-25T19:34:05.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.269+0000 7fc2fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:05.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.270+0000 7fc2fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc3041430c0 2024-03-25T19:34:05.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.270+0000 7fc2fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:05.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.278+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212669560 0 0) 0x7fc304135730 con 0x7fc30413f710 2024-03-25T19:34:05.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.278+0000 7fc2fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:05.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.278+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:05.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.278+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:05.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.278+0000 7fc2fa7fc640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:34:05.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.278+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e8004eb0 con 0x7fc30413f710 2024-03-25T19:34:05.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.280+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506321078 0 0) 0x7fc304135a50 con 0x7fc3041430c0 2024-03-25T19:34:05.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.280+0000 7fc2fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:05.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.280+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:05.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.280+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:05.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.280+0000 7fc2fa7fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:34:05.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.280+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc304135730 con 0x7fc3041430c0 2024-03-25T19:34:05.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.281+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72135879 0 0) 0x7fc3041358c0 con 0x7fc3040a98f0 2024-03-25T19:34:05.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.281+0000 7fc2fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:05.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.281+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:05.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.281+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:05.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.281+0000 7fc2fa7fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:34:05.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.281+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc304135a50 con 0x7fc3040a98f0 2024-03-25T19:34:05.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.289+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3399457896 0 0) 0x7fc2e8004eb0 con 0x7fc30413f710 2024-03-25T19:34:05.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.289+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3041358c0 con 0x7fc30413f710 2024-03-25T19:34:05.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.289+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc2f4001bc0 con 0x7fc30413f710 2024-03-25T19:34:05.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.289+0000 7fc2fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.289+0000 7fc2fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.289+0000 7fc2fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.289+0000 7fc2fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.296+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2661216033 0 0) 0x7fc304135730 con 0x7fc3041430c0 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.296+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2e8004eb0 con 0x7fc3041430c0 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.296+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc2fc001b10 con 0x7fc3041430c0 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.296+0000 7fc2fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.296+0000 7fc2fa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.296+0000 7fc2fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.296 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:05.297 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:05.297 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:05.297 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:05.297 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:05.297 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.307+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3400982578 0 0) 0x7fc304135a50 con 0x7fc3040a98f0 2024-03-25T19:34:05.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.307+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc304135730 con 0x7fc3040a98f0 2024-03-25T19:34:05.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.307+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc300001930 con 0x7fc3040a98f0 2024-03-25T19:34:05.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.307+0000 7fc2fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:05.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.307+0000 7fc2fa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.307+0000 7fc2fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:05.308 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.308+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2949659177 0 0) 0x7fc2e8004eb0 con 0x7fc3041430c0 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 >> v1:172.21.15.159:6789/0 conn(0x7fc3040a98f0 legacy=0x7fc3041dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 >> v1:172.21.15.134:6789/0 conn(0x7fc30413f710 legacy=0x7fc3041d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 1 monclient: found mon.c 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:05.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3041d5000 con 0x7fc3041430c0 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc2fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 5 monclient: authenticate success, global_id 4466 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 10 monclient: _renew_subs 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3041d52b0 con 0x7fc3041430c0 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 10 monclient: _renew_subs 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30413c900 con 0x7fc3041430c0 2024-03-25T19:34:05.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.309+0000 7fc30c1d6640 1 librados: init done 2024-03-25T19:34:05.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.310+0000 7fc30c1d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:05.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.310+0000 7fc30c1d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:05.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.310+0000 7fc30c1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:05.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.310+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30413cbb0 con 0x7fc3041430c0 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.604+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2fc002000 con 0x7fc3041430c0 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.604+0000 7fc2fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.604+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc2fc003970 con 0x7fc3041430c0 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.604+0000 7fc2fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.604+0000 7fc2fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.604+0000 7fc2fa7fc640 10 monclient: dump: 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:05.605 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:05.606 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.604+0000 7fc2fbfff640 4 set_mon_vals no callback set 2024-03-25T19:34:05.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.605+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fc2fc003b90 con 0x7fc3041430c0 2024-03-25T19:34:05.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.606+0000 7fc309f72640 10 monclient: get_auth_request con 0x7fc2e806e160 auth_method 0 2024-03-25T19:34:05.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.606+0000 7fc2fa7fc640 10 client.4466.objecter ms_handle_connect 0x7fc2e806e160 2024-03-25T19:34:05.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fc2fc053640 con 0x7fc3041430c0 2024-03-25T19:34:05.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 10 client.4466.objecter ms_dispatch 0x7fc304000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:05.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:05.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:05.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T19:34:05.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2fc04f150 con 0x7fc3041430c0 2024-03-25T19:34:05.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:05.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.611+0000 7fc2fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:05.726 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:10.399 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.775 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:10.447 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.777 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:10.449 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.780 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:10.452 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.808 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:10.480 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.826+0000 7fc30c1d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:05.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.826+0000 7fc30c1d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:05.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.826+0000 7fc30c1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:05.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.826+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3041371c0 con 0x7fc3041430c0 2024-03-25T19:34:05.832 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:10.504 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.861 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:10.508 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.866+0000 7fc2fa7fc640 1 -- 172.21.15.134:0/4228898169 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fc2fc051080 con 0x7fc3041430c0 2024-03-25T19:34:05.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.866+0000 7fc2fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:05.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.866+0000 7fc2fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:05.867 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:05.867 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:05.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 10 librados: watch_flush enter 2024-03-25T19:34:05.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 10 librados: watch_flush exit 2024-03-25T19:34:05.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 10 client.4466.objecter successfully canceled tick 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 >> v1:172.21.15.159:6800/1875874887 conn(0x7fc2e806e160 legacy=0x7fc2e8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 10 monclient: shutdown 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 >> v1:172.21.15.134:6790/0 conn(0x7fc3041430c0 legacy=0x7fc3041d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 shutdown_connections 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 >> 172.21.15.134:0/4228898169 conn(0x7fc3040b7ea0 msgr2=0x7fc30414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 shutdown_connections 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 1 -- 172.21.15.134:0/4228898169 wait complete. 2024-03-25T19:34:05.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:05.869+0000 7fc30c1d6640 1 librados: shutdown 2024-03-25T19:34:05.871 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:10.518 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:05.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:05.901 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:10.548 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.688 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:12.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.804+0000 7f10ebcd3640 10 monclient: get_monmap_and_config 2024-03-25T19:34:12.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.804+0000 7f10ebcd3640 10 monclient: build_initial_monmap 2024-03-25T19:34:12.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.804+0000 7f10ebcd3640 10 monclient: monmap: 2024-03-25T19:34:12.804 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:12.805779+0000 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:12.805779+0000 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:12.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.805+0000 7f10ebcd3640 1 Processor -- start 2024-03-25T19:34:12.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.805+0000 7f10ebcd3640 1 -- start start 2024-03-25T19:34:12.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.805+0000 7f10ebcd3640 10 monclient: init 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 10 monclient(hunting): picked mon.noname-c con 0x7f10e40b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 10 monclient(hunting): picked mon.noname-b con 0x7f10e40b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 10 monclient(hunting): picked mon.noname-a con 0x7f10e414be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e40af940 con 0x7f10e414be90 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e40b0a70 con 0x7f10e40b4d60 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e40b0d50 con 0x7f10e40b1730 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:12.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.806+0000 7f10ebcd3640 10 monclient(hunting): authenticate will time out at 6284.965465s 2024-03-25T19:34:12.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.818+0000 7f10e9a6f640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f10e40b4d60 0x7f10e413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:53154/0 (socket says 172.21.15.134:53154) 2024-03-25T19:34:12.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.819+0000 7f10e9a6f640 1 -- 172.21.15.134:0/3048894754 learned_addr learned my addr 172.21.15.134:0/3048894754 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:12.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.834+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508694332 0 0) 0x7f10e40b0a70 con 0x7f10e40b4d60 2024-03-25T19:34:12.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.834+0000 7f10e8a6d640 10 monclient(hunting): _negotiate 2024-03-25T19:34:12.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.834+0000 7f10e8a6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:12.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.834+0000 7f10e8a6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:12.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.834+0000 7f10e8a6d640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:34:12.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.834+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d0003520 con 0x7f10e40b4d60 2024-03-25T19:34:12.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.837+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487982587 0 0) 0x7f10e40af940 con 0x7f10e414be90 2024-03-25T19:34:12.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.837+0000 7f10e8a6d640 10 monclient(hunting): _negotiate 2024-03-25T19:34:12.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.837+0000 7f10e8a6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:12.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.837+0000 7f10e8a6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:12.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.837+0000 7f10e8a6d640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:34:12.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.837+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e40b0a70 con 0x7f10e414be90 2024-03-25T19:34:12.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.850+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763403893 0 0) 0x7f10e40b0d50 con 0x7f10e40b1730 2024-03-25T19:34:12.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.850+0000 7f10e8a6d640 10 monclient(hunting): _negotiate 2024-03-25T19:34:12.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.850+0000 7f10e8a6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:12.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.850+0000 7f10e8a6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:12.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.850+0000 7f10e8a6d640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:34:12.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.850+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e40af940 con 0x7f10e40b1730 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570867163 0 0) 0x7f10d0003520 con 0x7f10e40b4d60 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 >> v1:172.21.15.134:6790/0 conn(0x7f10e40b1730 legacy=0x7f10e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 >> v1:172.21.15.134:6789/0 conn(0x7f10e414be90 legacy=0x7f10e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 10 monclient: _finish_hunting 0 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 1 monclient: found mon.noname-c 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e40b1030 con 0x7f10e40b4d60 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 10 monclient: _finish_auth 0 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 10 monclient: _check_auth_tickets 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:12.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10ebcd3640 5 monclient: authenticate success, global_id 4462 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10ebcd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f10e0001690 con 0x7f10e40b4d60 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10e8a6d640 10 monclient: dump: 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:12.856 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:12.857 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:12.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.855+0000 7f10ebcd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:12.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10e00018b0 con 0x7f10e40b4d60 2024-03-25T19:34:12.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10e8a6d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:12.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10e8a6d640 1 -- 172.21.15.134:0/3048894754 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f10e0001c40 con 0x7f10e40b4d60 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10ebcd3640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10ebcd3640 4 set_mon_vals no callback set 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10e8a6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10e8a6d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.866+0000 7f10e8a6d640 10 monclient: dump: 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:12.867 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.867+0000 7f10ebcd3640 10 monclient: shutdown 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.867+0000 7f10ebcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.867+0000 7f10ebcd3640 1 -- 172.21.15.134:0/3048894754 >> v1:172.21.15.159:6789/0 conn(0x7f10e40b4d60 legacy=0x7f10e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.867+0000 7f10ebcd3640 1 -- 172.21.15.134:0/3048894754 shutdown_connections 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.867+0000 7f10ebcd3640 1 -- 172.21.15.134:0/3048894754 >> 172.21.15.134:0/3048894754 conn(0x7f10e40617a0 msgr2=0x7f10e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.868+0000 7f10ebcd3640 1 -- 172.21.15.134:0/3048894754 shutdown_connections 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.868+0000 7f10ebcd3640 1 -- 172.21.15.134:0/3048894754 wait complete. 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.868+0000 7f10ebcd3640 10 monclient: build_initial_monmap 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.868+0000 7f10ebcd3640 10 monclient: monmap: 2024-03-25T19:34:12.869 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 librados: starting msgr at 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 librados: starting objecter 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 Processor -- start 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 -- start start 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 librados: setting wanted keys 2024-03-25T19:34:12.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 librados: calling monclient init 2024-03-25T19:34:12.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 10 monclient: init 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 10 monclient(hunting): picked mon.noname-b con 0x7f10e40b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 10 monclient(hunting): picked mon.noname-a con 0x7f10e40b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 10 monclient(hunting): picked mon.noname-c con 0x7f10e414be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e4140d00 con 0x7f10e40b4d60 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.869+0000 7f10ebcd3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e4141cc0 con 0x7f10e40b1730 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.870+0000 7f10ebcd3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e41d8470 con 0x7f10e414be90 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.870+0000 7f10ebcd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:12.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.870+0000 7f10ebcd3640 10 monclient(hunting): authenticate will time out at 6285.028646s 2024-03-25T19:34:12.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.881+0000 7f10e926e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f10e40b4d60 0x7f10e41d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:56602/0 (socket says 172.21.15.134:56602) 2024-03-25T19:34:12.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.881+0000 7f10e926e640 1 -- 172.21.15.134:0/1434735134 learned_addr learned my addr 172.21.15.134:0/1434735134 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:12.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.881+0000 7f10da7fc640 10 client.?.objecter ms_handle_connect 0x7f10e40b4d60 2024-03-25T19:34:12.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.881+0000 7f10da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:12.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.894+0000 7f10da7fc640 10 client.?.objecter ms_handle_connect 0x7f10e414be90 2024-03-25T19:34:12.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.894+0000 7f10da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:12.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.894+0000 7f10da7fc640 10 client.?.objecter ms_handle_connect 0x7f10e40b1730 2024-03-25T19:34:12.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.894+0000 7f10da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:12.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.897+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803912465 0 0) 0x7f10e41d8470 con 0x7f10e414be90 2024-03-25T19:34:12.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.897+0000 7f10da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:12.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.897+0000 7f10da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:12.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.897+0000 7f10da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:12.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.897+0000 7f10da7fc640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:34:12.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.897+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c8004eb0 con 0x7f10e414be90 2024-03-25T19:34:12.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.902+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116051499 0 0) 0x7f10e4140d00 con 0x7f10e40b4d60 2024-03-25T19:34:12.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.902+0000 7f10da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:12.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.902+0000 7f10da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:12.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.902+0000 7f10da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:12.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.902+0000 7f10da7fc640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:34:12.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.902+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e41d8470 con 0x7f10e40b4d60 2024-03-25T19:34:12.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.903+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189791197 0 0) 0x7f10c8004eb0 con 0x7f10e414be90 2024-03-25T19:34:12.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.903+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10e4140d00 con 0x7f10e414be90 2024-03-25T19:34:12.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.903+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f10dc001bd0 con 0x7f10e414be90 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.903+0000 7f10da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.903+0000 7f10da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.903+0000 7f10da7fc640 10 monclient(hunting): dump: 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:12.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.903+0000 7f10da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:12.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.905+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914489508 0 0) 0x7f10e4141cc0 con 0x7f10e40b1730 2024-03-25T19:34:12.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.905+0000 7f10da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:12.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.905+0000 7f10da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:12.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.905+0000 7f10da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:12.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.905+0000 7f10da7fc640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:34:12.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.905+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c8004eb0 con 0x7f10e40b1730 2024-03-25T19:34:12.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.912+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260349187 0 0) 0x7f10e41d8470 con 0x7f10e40b4d60 2024-03-25T19:34:12.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.912+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10c8005b00 con 0x7f10e40b4d60 2024-03-25T19:34:12.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.912+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f10d40019f0 con 0x7f10e40b4d60 2024-03-25T19:34:12.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.912+0000 7f10da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:12.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.912+0000 7f10da7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.912+0000 7f10da7fc640 10 monclient(hunting): dump: 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:12.913 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2594470600 0 0) 0x7f10e4140d00 con 0x7f10e414be90 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 >> v1:172.21.15.159:6789/0 conn(0x7f10e40b1730 legacy=0x7f10e41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 >> v1:172.21.15.134:6789/0 conn(0x7f10e40b4d60 legacy=0x7f10e41d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 1 monclient: found mon.c 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:12.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e41d8600 con 0x7f10e414be90 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 5 monclient: authenticate success, global_id 4472 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 10 monclient: _renew_subs 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e413b3c0 con 0x7f10e414be90 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 10 monclient: _renew_subs 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e41425a0 con 0x7f10e414be90 2024-03-25T19:34:12.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.916+0000 7f10ebcd3640 1 librados: init done 2024-03-25T19:34:12.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.917+0000 7f10ebcd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:12.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.917+0000 7f10ebcd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:12.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.917+0000 7f10ebcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:12.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:12.917+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e40b2270 con 0x7f10e414be90 2024-03-25T19:34:13.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.235+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10dc003970 con 0x7f10e414be90 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.235+0000 7f10da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.235+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f10dc002100 con 0x7f10e414be90 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.235+0000 7f10da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.235+0000 7f10da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.235+0000 7f10da7fc640 10 monclient: dump: 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:13.235 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:13.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.235+0000 7f10dbfff640 4 set_mon_vals no callback set 2024-03-25T19:34:13.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.236+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f10dc019b70 con 0x7f10e414be90 2024-03-25T19:34:13.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.236+0000 7f10e9a6f640 10 monclient: get_auth_request con 0x7f10c806e120 auth_method 0 2024-03-25T19:34:13.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.236+0000 7f10da7fc640 10 client.4472.objecter ms_handle_connect 0x7f10c806e120 2024-03-25T19:34:13.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f10dc053870 con 0x7f10e414be90 2024-03-25T19:34:13.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 10 client.4472.objecter ms_dispatch 0x7f10e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:13.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:13.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:13.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-25T19:34:13.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10dc04f1b0 con 0x7f10e414be90 2024-03-25T19:34:13.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:13.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.240+0000 7f10da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:13.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.494+0000 7f10ebcd3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:13.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.494+0000 7f10ebcd3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:13.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.494+0000 7f10ebcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:13.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.494+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10e40627b0 con 0x7f10e414be90 2024-03-25T19:34:13.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.527+0000 7f10da7fc640 1 -- 172.21.15.134:0/1434735134 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f10dc0512b0 con 0x7f10e414be90 2024-03-25T19:34:13.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.527+0000 7f10da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:13.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.527+0000 7f10da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:13.528 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:13.528 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:13.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 10 librados: watch_flush enter 2024-03-25T19:34:13.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 10 librados: watch_flush exit 2024-03-25T19:34:13.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 10 client.4472.objecter successfully canceled tick 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 >> v1:172.21.15.159:6800/1875874887 conn(0x7f10c806e120 legacy=0x7f10c80705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 10 monclient: shutdown 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 >> v1:172.21.15.134:6790/0 conn(0x7f10e414be90 legacy=0x7f10e4134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 shutdown_connections 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 >> 172.21.15.134:0/1434735134 conn(0x7f10e40617a0 msgr2=0x7f10e40b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 shutdown_connections 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 1 -- 172.21.15.134:0/1434735134 wait complete. 2024-03-25T19:34:13.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:13.530+0000 7f10ebcd3640 1 librados: shutdown 2024-03-25T19:34:13.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:20.350 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:20.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.453+0000 7f5240910640 10 monclient: get_monmap_and_config 2024-03-25T19:34:20.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.453+0000 7f5240910640 10 monclient: build_initial_monmap 2024-03-25T19:34:20.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.453+0000 7f5240910640 10 monclient: monmap: 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:20.454726+0000 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:20.454726+0000 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:20.454 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.454+0000 7f5240910640 1 Processor -- start 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.454+0000 7f5240910640 1 -- start start 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.454+0000 7f5240910640 10 monclient: init 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 10 monclient(hunting): picked mon.noname-a con 0x7f523c0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 10 monclient(hunting): picked mon.noname-c con 0x7f523c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 10 monclient(hunting): picked mon.noname-b con 0x7f523c14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0af940 con 0x7f523c0b4d60 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0b0a70 con 0x7f523c0b1730 2024-03-25T19:34:20.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0b0d50 con 0x7f523c14be90 2024-03-25T19:34:20.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:20.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5240910640 10 monclient(hunting): authenticate will time out at 6292.613927s 2024-03-25T19:34:20.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5231d9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f523c0b1730 0x7f523c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58034/0 (socket says 172.21.15.134:58034) 2024-03-25T19:34:20.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.455+0000 7f5231d9b640 1 -- 172.21.15.134:0/3861625530 learned_addr learned my addr 172.21.15.134:0/3861625530 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:20.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.499+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092112970 0 0) 0x7f523c0af940 con 0x7f523c0b4d60 2024-03-25T19:34:20.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.499+0000 7f5239d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:20.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.499+0000 7f5239d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:20.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.499+0000 7f5239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:20.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.499+0000 7f5239d9b640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:34:20.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.499+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5228003520 con 0x7f523c0b4d60 2024-03-25T19:34:20.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.532+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847648107 0 0) 0x7f523c0b0a70 con 0x7f523c0b1730 2024-03-25T19:34:20.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.532+0000 7f5239d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:20.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.532+0000 7f5239d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:20.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.532+0000 7f5239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:20.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.532+0000 7f5239d9b640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:34:20.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.532+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523c0af940 con 0x7f523c0b1730 2024-03-25T19:34:20.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3028885296 0 0) 0x7f5228003520 con 0x7f523c0b4d60 2024-03-25T19:34:20.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 >> v1:172.21.15.134:6790/0 conn(0x7f523c14be90 legacy=0x7f523c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:20.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 >> v1:172.21.15.159:6789/0 conn(0x7f523c0b1730 legacy=0x7f523c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:20.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:34:20.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 1 monclient: found mon.noname-a 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0b1030 con 0x7f523c0b4d60 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 10 monclient: _finish_auth 0 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f522c001690 con 0x7f523c0b4d60 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5239d9b640 10 monclient: dump: 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:20.543 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:20.544 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:20.544 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:20.544 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:20.544 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:20.544 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:20.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5240910640 5 monclient: authenticate success, global_id 4509 2024-03-25T19:34:20.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.542+0000 7f5240910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:20.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.548+0000 7f5239d9b640 1 -- 172.21.15.134:0/3861625530 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f522c0017f0 con 0x7f523c0b4d60 2024-03-25T19:34:20.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.548+0000 7f5239d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:20.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.548+0000 7f5240910640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:20.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.548+0000 7f5240910640 4 set_mon_vals no callback set 2024-03-25T19:34:20.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.549+0000 7f5240910640 10 monclient: shutdown 2024-03-25T19:34:20.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.549+0000 7f5240910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:20.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.549+0000 7f5240910640 1 -- 172.21.15.134:0/3861625530 >> v1:172.21.15.134:6789/0 conn(0x7f523c0b4d60 legacy=0x7f523c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:20.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.549+0000 7f5240910640 1 -- 172.21.15.134:0/3861625530 shutdown_connections 2024-03-25T19:34:20.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.549+0000 7f5240910640 1 -- 172.21.15.134:0/3861625530 >> 172.21.15.134:0/3861625530 conn(0x7f523c0617a0 msgr2=0x7f523c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:20.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.549+0000 7f5240910640 1 -- 172.21.15.134:0/3861625530 shutdown_connections 2024-03-25T19:34:20.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.549+0000 7f5240910640 1 -- 172.21.15.134:0/3861625530 wait complete. 2024-03-25T19:34:20.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.550+0000 7f5240910640 10 monclient: build_initial_monmap 2024-03-25T19:34:20.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.550+0000 7f5240910640 10 monclient: monmap: 2024-03-25T19:34:20.550 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.550+0000 7f5240910640 1 librados: starting msgr at 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.550+0000 7f5240910640 1 librados: starting objecter 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.550+0000 7f5240910640 1 Processor -- start 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 1 -- start start 2024-03-25T19:34:20.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 1 librados: setting wanted keys 2024-03-25T19:34:20.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 1 librados: calling monclient init 2024-03-25T19:34:20.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient: init 2024-03-25T19:34:20.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient(hunting): picked mon.noname-c con 0x7f523c0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient(hunting): picked mon.noname-b con 0x7f523c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient(hunting): picked mon.noname-a con 0x7f523c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c1376f0 con 0x7f523c14be90 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c13a540 con 0x7f523c0b4d60 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0aa8d0 con 0x7f523c0b1730 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.551+0000 7f5240910640 10 monclient(hunting): authenticate will time out at 6292.710391s 2024-03-25T19:34:20.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.554+0000 7f5231d9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f523c0b4d60 0x7f523c1368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58086/0 (socket says 172.21.15.134:58086) 2024-03-25T19:34:20.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.554+0000 7f5231d9b640 1 -- 172.21.15.134:0/3227622174 learned_addr learned my addr 172.21.15.134:0/3227622174 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:20.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.554+0000 7f52337fe640 10 client.?.objecter ms_handle_connect 0x7f523c0b4d60 2024-03-25T19:34:20.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.555+0000 7f52337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:20.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.559+0000 7f52337fe640 10 client.?.objecter ms_handle_connect 0x7f523c0b1730 2024-03-25T19:34:20.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.559+0000 7f52337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:20.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.570+0000 7f52337fe640 10 client.?.objecter ms_handle_connect 0x7f523c14be90 2024-03-25T19:34:20.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.570+0000 7f52337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:20.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.630+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464728203 0 0) 0x7f523c1376f0 con 0x7f523c14be90 2024-03-25T19:34:20.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.630+0000 7f52337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:20.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.630+0000 7f52337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:20.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.630+0000 7f52337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:20.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.630+0000 7f52337fe640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:34:20.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.630+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c004eb0 con 0x7f523c14be90 2024-03-25T19:34:20.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.638+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905811909 0 0) 0x7f523c0aa8d0 con 0x7f523c0b1730 2024-03-25T19:34:20.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.638+0000 7f52337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:20.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.638+0000 7f52337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:20.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.638+0000 7f52337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:20.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.638+0000 7f52337fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:34:20.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.638+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523c1376f0 con 0x7f523c0b1730 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.646+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641322121 0 0) 0x7f521c004eb0 con 0x7f523c14be90 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.646+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f523c0aa8d0 con 0x7f523c14be90 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.646+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f5234001b20 con 0x7f523c14be90 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.646+0000 7f52337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.646+0000 7f52337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.646+0000 7f52337fe640 10 monclient(hunting): dump: 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:20.647 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:20.648 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:20.648 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:20.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.646+0000 7f52337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:20.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.658+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 378567118 0 0) 0x7f523c0aa8d0 con 0x7f523c14be90 2024-03-25T19:34:20.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 >> v1:172.21.15.134:6790/0 conn(0x7f523c0b1730 legacy=0x7f523c139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:20.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 >> v1:172.21.15.159:6789/0 conn(0x7f523c0b4d60 legacy=0x7f523c1368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:20.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:20.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 1 monclient: found mon.a 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0aabb0 con 0x7f523c14be90 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f52337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 10 monclient: _renew_subs 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c1387c0 con 0x7f523c14be90 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 10 monclient: _renew_subs 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c13b0b0 con 0x7f523c14be90 2024-03-25T19:34:20.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.659+0000 7f5240910640 1 librados: init done 2024-03-25T19:34:20.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.660+0000 7f5240910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:20.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.660+0000 7f5240910640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:20.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.660+0000 7f5240910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:34:20.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.660+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c0627b0 con 0x7f523c14be90 2024-03-25T19:34:20.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.974+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5234002030 con 0x7f523c14be90 2024-03-25T19:34:20.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.974+0000 7f52337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:20.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.974+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f5234003b10 con 0x7f523c14be90 2024-03-25T19:34:20.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.974+0000 7f52337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.974+0000 7f52337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.974+0000 7f52337fe640 10 monclient: dump: 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:20.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.974+0000 7f523959a640 4 set_mon_vals no callback set 2024-03-25T19:34:20.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.975+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f5234003d30 con 0x7f523c14be90 2024-03-25T19:34:20.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.976+0000 7f523a59c640 10 monclient: get_auth_request con 0x7f521c06dd00 auth_method 0 2024-03-25T19:34:20.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.976+0000 7f52337fe640 10 client.4512.objecter ms_handle_connect 0x7f521c06dd00 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f5234053750 con 0x7f523c14be90 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 10 client.4512.objecter ms_dispatch 0x7f523c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f523404f260 con 0x7f523c14be90 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:20.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:20.985+0000 7f52337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:21.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.236+0000 7f5240910640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:21.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.236+0000 7f5240910640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:21.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.236+0000 7f5240910640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:34:21.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.236+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f523c069f40 con 0x7f523c14be90 2024-03-25T19:34:21.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.263+0000 7f52337fe640 1 -- 172.21.15.134:0/3227622174 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f5234051190 con 0x7f523c14be90 2024-03-25T19:34:21.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.263+0000 7f52337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:21.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.263+0000 7f52337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:21.263 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:21.264 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:21.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 10 librados: watch_flush enter 2024-03-25T19:34:21.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 10 librados: watch_flush exit 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 10 client.4512.objecter successfully canceled tick 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 >> v1:172.21.15.159:6800/1875874887 conn(0x7f521c06dd00 legacy=0x7f521c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 10 monclient: shutdown 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 >> v1:172.21.15.134:6789/0 conn(0x7f523c14be90 legacy=0x7f523c136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 shutdown_connections 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 >> 172.21.15.134:0/3227622174 conn(0x7f523c0617a0 msgr2=0x7f523c145290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 shutdown_connections 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 1 -- 172.21.15.134:0/3227622174 wait complete. 2024-03-25T19:34:21.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:21.266+0000 7f5240910640 1 librados: shutdown 2024-03-25T19:34:21.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:25.608 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:30.280 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.619 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:30.291 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.660 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:30.333 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.662 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:30.334 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.674 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:30.346 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.681 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:30.327 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.690 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:30.337 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.697 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:30.369 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:25.708 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:30.354 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:28.085 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:28.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.205+0000 7f33e2b5d640 10 monclient: get_monmap_and_config 2024-03-25T19:34:28.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.205+0000 7f33e2b5d640 10 monclient: build_initial_monmap 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.205+0000 7f33e2b5d640 10 monclient: monmap: 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:28.206990+0000 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:28.206990+0000 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:28.206 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:28.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.207+0000 7f33e2b5d640 1 Processor -- start 2024-03-25T19:34:28.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.207+0000 7f33e2b5d640 1 -- start start 2024-03-25T19:34:28.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.207+0000 7f33e2b5d640 10 monclient: init 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.207+0000 7f33e2b5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.207+0000 7f33e2b5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.207+0000 7f33e2b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f33dc0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.208+0000 7f33e2b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f33dc13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.208+0000 7f33e2b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f33dc1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.208+0000 7f33e2b5d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc142520 con 0x7f33dc0a98f0 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.208+0000 7f33e2b5d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc0b28a0 con 0x7f33dc13f710 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.208+0000 7f33e2b5d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc0b2a30 con 0x7f33dc1430c0 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.208+0000 7f33e2b5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:28.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.208+0000 7f33e2b5d640 10 monclient(hunting): authenticate will time out at 6300.366811s 2024-03-25T19:34:28.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.209+0000 7f33dbfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f33dc13f710 0x7f33dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:50730/0 (socket says 172.21.15.134:50730) 2024-03-25T19:34:28.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.209+0000 7f33dbfff640 1 -- 172.21.15.134:0/1611014931 learned_addr learned my addr 172.21.15.134:0/1611014931 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:28.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.293+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986972620 0 0) 0x7f33dc142520 con 0x7f33dc0a98f0 2024-03-25T19:34:28.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.293+0000 7f33db7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:28.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.293+0000 7f33db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:28.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.293+0000 7f33db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:28.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.293+0000 7f33db7fe640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:34:28.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.293+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc003520 con 0x7f33dc0a98f0 2024-03-25T19:34:28.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2438835927 0 0) 0x7f33cc003520 con 0x7f33dc0a98f0 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 >> v1:172.21.15.134:6790/0 conn(0x7f33dc1430c0 legacy=0x7f33dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 >> v1:172.21.15.159:6789/0 conn(0x7f33dc13f710 legacy=0x7f33dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 1 monclient: found mon.noname-a 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc0b44c0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f33d0001690 con 0x7f33dc0a98f0 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33db7fe640 10 monclient: dump: 2024-03-25T19:34:28.315 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33e2b5d640 5 monclient: authenticate success, global_id 4515 2024-03-25T19:34:28.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.314+0000 7f33e2b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33d00017f0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33db7fe640 1 -- 172.21.15.134:0/1611014931 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f33d0001c00 con 0x7f33dc0a98f0 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33e2b5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33e2b5d640 4 set_mon_vals no callback set 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.328+0000 7f33db7fe640 10 monclient: dump: 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:28.328 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.329 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.329 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:28.329 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:28.329 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:28.329 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:28.329 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:28.329 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 10 monclient: shutdown 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 1 -- 172.21.15.134:0/1611014931 >> v1:172.21.15.134:6789/0 conn(0x7f33dc0a98f0 legacy=0x7f33dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 1 -- 172.21.15.134:0/1611014931 shutdown_connections 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 1 -- 172.21.15.134:0/1611014931 >> 172.21.15.134:0/1611014931 conn(0x7f33dc0b7ea0 msgr2=0x7f33dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 1 -- 172.21.15.134:0/1611014931 shutdown_connections 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 1 -- 172.21.15.134:0/1611014931 wait complete. 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.329+0000 7f33e2b5d640 10 monclient: build_initial_monmap 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 10 monclient: monmap: 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:28.331 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 1 librados: starting msgr at 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 1 librados: starting objecter 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 1 Processor -- start 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 1 -- start start 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 1 librados: setting wanted keys 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 1 librados: calling monclient init 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.330+0000 7f33e2b5d640 10 monclient: init 2024-03-25T19:34:28.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f33dc0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f33dc13f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f33dc1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc138c70 con 0x7f33dc13f710 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc1367d0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc136960 con 0x7f33dc1430c0 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:28.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.331+0000 7f33e2b5d640 10 monclient(hunting): authenticate will time out at 6300.489906s 2024-03-25T19:34:28.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.340+0000 7f33e08f9640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f33dc0a98f0 0x7f33dc137610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:50784/0 (socket says 172.21.15.134:50784) 2024-03-25T19:34:28.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.340+0000 7f33e08f9640 1 -- 172.21.15.134:0/4060792893 learned_addr learned my addr 172.21.15.134:0/4060792893 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:28.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.341+0000 7f33d97fa640 10 client.?.objecter ms_handle_connect 0x7f33dc0a98f0 2024-03-25T19:34:28.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.341+0000 7f33d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:28.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.347+0000 7f33d97fa640 10 client.?.objecter ms_handle_connect 0x7f33dc13f710 2024-03-25T19:34:28.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.347+0000 7f33d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:28.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.351+0000 7f33d97fa640 10 client.?.objecter ms_handle_connect 0x7f33dc1430c0 2024-03-25T19:34:28.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.351+0000 7f33d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:28.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.355+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534994867 0 0) 0x7f33dc1367d0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.355+0000 7f33d97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:28.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.355+0000 7f33d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:28.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.355+0000 7f33d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:28.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.355+0000 7f33d97fa640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:34:28.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.355+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c0004eb0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.359+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681046157 0 0) 0x7f33dc138c70 con 0x7f33dc13f710 2024-03-25T19:34:28.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.359+0000 7f33d97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:28.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.359+0000 7f33d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:28.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.359+0000 7f33d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:28.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.359+0000 7f33d97fa640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:34:28.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.359+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc1367d0 con 0x7f33dc13f710 2024-03-25T19:34:28.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.364+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644549935 0 0) 0x7f33dc136960 con 0x7f33dc1430c0 2024-03-25T19:34:28.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.364+0000 7f33d97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:28.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.364+0000 7f33d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:28.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.364+0000 7f33d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:28.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.364+0000 7f33d97fa640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:34:28.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.364+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc138c70 con 0x7f33dc1430c0 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.367+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2738837407 0 0) 0x7f33c0004eb0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.367+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33dc136960 con 0x7f33dc0a98f0 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.367+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f33d0001690 con 0x7f33dc0a98f0 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.367+0000 7f33d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.367+0000 7f33d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.367+0000 7f33d97fa640 10 monclient(hunting): dump: 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:28.367 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:28.368 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:28.368 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:28.368 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:28.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.367+0000 7f33d97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.377+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1376103576 0 0) 0x7f33dc1367d0 con 0x7f33dc13f710 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.377+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33c0004eb0 con 0x7f33dc13f710 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.377+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f33c80018a0 con 0x7f33dc13f710 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.377+0000 7f33d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.377+0000 7f33d97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.377+0000 7f33d97fa640 10 monclient(hunting): dump: 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:28.377 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1090451051 0 0) 0x7f33dc136960 con 0x7f33dc0a98f0 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 >> v1:172.21.15.134:6790/0 conn(0x7f33dc1430c0 legacy=0x7f33dc1e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 >> v1:172.21.15.134:6789/0 conn(0x7f33dc13f710 legacy=0x7f33dc132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 1 monclient: found mon.b 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc139630 con 0x7f33dc0a98f0 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:28.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 5 monclient: authenticate success, global_id 4477 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 10 monclient: _renew_subs 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc139880 con 0x7f33dc0a98f0 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 10 monclient: _renew_subs 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc1337f0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.383+0000 7f33e2b5d640 1 librados: init done 2024-03-25T19:34:28.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.384+0000 7f33e2b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:28.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.384+0000 7f33e2b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:28.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.384+0000 7f33e2b5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:28.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.384+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc133aa0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.662+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33d00038a0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.662+0000 7f33d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.662+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f33d0003cf0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.662+0000 7f33d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.662+0000 7f33d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.662+0000 7f33d97fa640 10 monclient: dump: 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:28.662 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.662+0000 7f33daffd640 4 set_mon_vals no callback set 2024-03-25T19:34:28.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.663+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f33d0003f10 con 0x7f33dc0a98f0 2024-03-25T19:34:28.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.663+0000 7f33dbfff640 10 monclient: get_auth_request con 0x7f33c006df70 auth_method 0 2024-03-25T19:34:28.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.663+0000 7f33d97fa640 10 client.4477.objecter ms_handle_connect 0x7f33c006df70 2024-03-25T19:34:28.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f33d00549a0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 10 client.4477.objecter ms_dispatch 0x7f33dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:28.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:28.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:28.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T19:34:28.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33d00504b0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:28.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.667+0000 7f33d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:28.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.925+0000 7f33e2b5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:28.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.925+0000 7f33e2b5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:28.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.925+0000 7f33e2b5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:28.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.925+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33dc074cc0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.950+0000 7f33d97fa640 1 -- 172.21.15.134:0/4060792893 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f33d00523e0 con 0x7f33dc0a98f0 2024-03-25T19:34:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.950+0000 7f33d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.950+0000 7f33d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:28.950 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:28.950 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.952+0000 7f33e2b5d640 10 librados: watch_flush enter 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.952+0000 7f33e2b5d640 10 librados: watch_flush exit 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.952+0000 7f33e2b5d640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.952+0000 7f33e2b5d640 10 client.4477.objecter successfully canceled tick 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 >> v1:172.21.15.159:6800/1875874887 conn(0x7f33c006df70 legacy=0x7f33c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 10 monclient: shutdown 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 >> v1:172.21.15.159:6789/0 conn(0x7f33dc0a98f0 legacy=0x7f33dc137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:28.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 shutdown_connections 2024-03-25T19:34:28.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 >> 172.21.15.134:0/4060792893 conn(0x7f33dc0b7ea0 msgr2=0x7f33dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:28.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 shutdown_connections 2024-03-25T19:34:28.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 1 -- 172.21.15.134:0/4060792893 wait complete. 2024-03-25T19:34:28.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:28.953+0000 7f33e2b5d640 1 librados: shutdown 2024-03-25T19:34:28.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:35.772 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:35.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.889+0000 7f946e3b6640 10 monclient: get_monmap_and_config 2024-03-25T19:34:35.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.889+0000 7f946e3b6640 10 monclient: build_initial_monmap 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.889+0000 7f946e3b6640 10 monclient: monmap: 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:35.890588+0000 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:35.890588+0000 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:35.890 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.890+0000 7f946e3b6640 1 Processor -- start 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.890+0000 7f946e3b6640 1 -- start start 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.890+0000 7f946e3b6640 10 monclient: init 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.891+0000 7f946e3b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.891+0000 7f946e3b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.891+0000 7f946e3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f94680a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.891+0000 7f946e3b6640 10 monclient(hunting): picked mon.noname-c con 0x7f946813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.891+0000 7f946e3b6640 10 monclient(hunting): picked mon.noname-a con 0x7f94681430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:35.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.891+0000 7f946e3b6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9468142520 con 0x7f94681430c0 2024-03-25T19:34:35.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.891+0000 7f946e3b6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94680b28a0 con 0x7f946813f710 2024-03-25T19:34:35.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.892+0000 7f946e3b6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94680b2a30 con 0x7f94680a98f0 2024-03-25T19:34:35.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.892+0000 7f946e3b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:35.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.892+0000 7f946e3b6640 10 monclient(hunting): authenticate will time out at 6308.050641s 2024-03-25T19:34:35.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.897+0000 7f94677fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f946813f710 0x7f94681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:47604/0 (socket says 172.21.15.134:47604) 2024-03-25T19:34:35.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.897+0000 7f94677fe640 1 -- 172.21.15.134:0/1351794741 learned_addr learned my addr 172.21.15.134:0/1351794741 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:35.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.920+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994182447 0 0) 0x7f94680b28a0 con 0x7f946813f710 2024-03-25T19:34:35.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.920+0000 7f9466ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:35.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.920+0000 7f9466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:35.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.920+0000 7f9466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:35.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.920+0000 7f9466ffd640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:34:35.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.920+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9450003520 con 0x7f946813f710 2024-03-25T19:34:35.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.927+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972853669 0 0) 0x7f9468142520 con 0x7f94681430c0 2024-03-25T19:34:35.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.927+0000 7f9466ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:35.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.927+0000 7f9466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:35.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.927+0000 7f9466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:35.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.927+0000 7f9466ffd640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:34:35.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.927+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94680b28a0 con 0x7f94681430c0 2024-03-25T19:34:35.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393420612 0 0) 0x7f9450003520 con 0x7f946813f710 2024-03-25T19:34:35.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 >> v1:172.21.15.134:6790/0 conn(0x7f94680a98f0 legacy=0x7f94680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:35.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 >> v1:172.21.15.134:6789/0 conn(0x7f94681430c0 legacy=0x7f946814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:35.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:35.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 1 monclient: found mon.noname-c 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94680b44c0 con 0x7f946813f710 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f945c001690 con 0x7f946813f710 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f9466ffd640 10 monclient: dump: 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:35.936 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:35.937 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:35.937 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:35.937 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:35.937 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:35.937 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:35.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f946e3b6640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:34:35.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.935+0000 7f946e3b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:35.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f945c0017f0 con 0x7f946813f710 2024-03-25T19:34:35.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f9466ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:35.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f9466ffd640 1 -- 172.21.15.134:0/1351794741 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f945c001c00 con 0x7f946813f710 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f946e3b6640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f946e3b6640 4 set_mon_vals no callback set 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f9466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f9466ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.953+0000 7f9466ffd640 10 monclient: dump: 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:35.954 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:35.955 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.954+0000 7f946e3b6640 10 monclient: shutdown 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.954+0000 7f946e3b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.954+0000 7f946e3b6640 1 -- 172.21.15.134:0/1351794741 >> v1:172.21.15.159:6789/0 conn(0x7f946813f710 legacy=0x7f94681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.954+0000 7f946e3b6640 1 -- 172.21.15.134:0/1351794741 shutdown_connections 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.954+0000 7f946e3b6640 1 -- 172.21.15.134:0/1351794741 >> 172.21.15.134:0/1351794741 conn(0x7f94680b7ea0 msgr2=0x7f94680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.955+0000 7f946e3b6640 1 -- 172.21.15.134:0/1351794741 shutdown_connections 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.955+0000 7f946e3b6640 1 -- 172.21.15.134:0/1351794741 wait complete. 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.955+0000 7f946e3b6640 10 monclient: build_initial_monmap 2024-03-25T19:34:35.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.955+0000 7f946e3b6640 10 monclient: monmap: 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.955+0000 7f946e3b6640 1 librados: starting msgr at 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 1 librados: starting objecter 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 1 Processor -- start 2024-03-25T19:34:35.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 1 -- start start 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 1 librados: setting wanted keys 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 1 librados: calling monclient init 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 10 monclient: init 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f94680a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.956+0000 7f946e3b6640 10 monclient(hunting): picked mon.noname-c con 0x7f946813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.957+0000 7f946e3b6640 10 monclient(hunting): picked mon.noname-a con 0x7f94681430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.957+0000 7f946e3b6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9468137f70 con 0x7f94681430c0 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.957+0000 7f946e3b6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9468139090 con 0x7f94680a98f0 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.957+0000 7f946e3b6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9468138400 con 0x7f946813f710 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.957+0000 7f946e3b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:35.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.957+0000 7f946e3b6640 10 monclient(hunting): authenticate will time out at 6308.115700s 2024-03-25T19:34:35.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.966+0000 7f946c953640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f94681430c0 0x7f9468132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:54158/0 (socket says 172.21.15.134:54158) 2024-03-25T19:34:35.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.966+0000 7f946c953640 1 -- 172.21.15.134:0/3324391717 learned_addr learned my addr 172.21.15.134:0/3324391717 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:35.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.973+0000 7f9464ff9640 10 client.?.objecter ms_handle_connect 0x7f94681430c0 2024-03-25T19:34:35.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.973+0000 7f9464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:35.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.975+0000 7f9464ff9640 10 client.?.objecter ms_handle_connect 0x7f94680a98f0 2024-03-25T19:34:35.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.975+0000 7f9464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:35.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.976+0000 7f9464ff9640 10 client.?.objecter ms_handle_connect 0x7f946813f710 2024-03-25T19:34:35.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.976+0000 7f9464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:35.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.982+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216632233 0 0) 0x7f9468139090 con 0x7f94680a98f0 2024-03-25T19:34:35.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.982+0000 7f9464ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:35.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.982+0000 7f9464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:35.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.982+0000 7f9464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:35.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.982+0000 7f9464ff9640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:34:35.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.982+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454004eb0 con 0x7f94680a98f0 2024-03-25T19:34:35.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.985+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895160326 0 0) 0x7f9468137f70 con 0x7f94681430c0 2024-03-25T19:34:35.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.985+0000 7f9464ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:35.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.985+0000 7f9464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:35.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.985+0000 7f9464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:35.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.985+0000 7f9464ff9640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:34:35.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.985+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9468139090 con 0x7f94681430c0 2024-03-25T19:34:35.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.991+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 387217637 0 0) 0x7f9454004eb0 con 0x7f94680a98f0 2024-03-25T19:34:35.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.991+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9468137f70 con 0x7f94680a98f0 2024-03-25T19:34:35.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.991+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f94580018c0 con 0x7f94680a98f0 2024-03-25T19:34:35.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.991+0000 7f9464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:35.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.991+0000 7f9464ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.991+0000 7f9464ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:35.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.991+0000 7f9464ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:35.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.996+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204263799 0 0) 0x7f9468138400 con 0x7f946813f710 2024-03-25T19:34:35.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.996+0000 7f9464ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:35.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.996+0000 7f9464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:35.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.996+0000 7f9464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:35.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.996+0000 7f9464ff9640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:34:35.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.996+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454004eb0 con 0x7f946813f710 2024-03-25T19:34:35.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 801662008 0 0) 0x7f9468137f70 con 0x7f94680a98f0 2024-03-25T19:34:35.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 >> v1:172.21.15.134:6790/0 conn(0x7f946813f710 legacy=0x7f94681d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:35.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 >> v1:172.21.15.134:6789/0 conn(0x7f94681430c0 legacy=0x7f9468132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:35.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 1 monclient: found mon.b 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468139300 con 0x7f94680a98f0 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 10 monclient: _finish_auth 0 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f9464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 10 monclient: _renew_subs 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94681d7df0 con 0x7f94680a98f0 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 10 monclient: _renew_subs 2024-03-25T19:34:35.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:36.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94680b76f0 con 0x7f94680a98f0 2024-03-25T19:34:36.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.998+0000 7f946e3b6640 1 librados: init done 2024-03-25T19:34:36.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.999+0000 7f946e3b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:36.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.999+0000 7f946e3b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:36.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.999+0000 7f946e3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:36.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:35.999+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94680b7a40 con 0x7f94680a98f0 2024-03-25T19:34:36.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.207+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94580035f0 con 0x7f94680a98f0 2024-03-25T19:34:36.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.207+0000 7f9464ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:36.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.207+0000 7f94667fc640 4 set_mon_vals no callback set 2024-03-25T19:34:36.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.278+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9458003a00 con 0x7f94680a98f0 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.278+0000 7f9464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.278+0000 7f9464ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.278+0000 7f9464ff9640 10 monclient: dump: 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:36.278 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:36.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.278+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f9458003be0 con 0x7f94680a98f0 2024-03-25T19:34:36.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.279+0000 7f94677fe640 10 monclient: get_auth_request con 0x7f945406ddb0 auth_method 0 2024-03-25T19:34:36.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.279+0000 7f9464ff9640 10 client.4483.objecter ms_handle_connect 0x7f945406ddb0 2024-03-25T19:34:36.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.282+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f9458053600 con 0x7f94680a98f0 2024-03-25T19:34:36.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.282+0000 7f9464ff9640 10 client.4483.objecter ms_dispatch 0x7f9468000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:36.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.282+0000 7f9464ff9640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:36.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.282+0000 7f9464ff9640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:36.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.282+0000 7f9464ff9640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T19:34:36.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.286+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f945804f110 con 0x7f94680a98f0 2024-03-25T19:34:36.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.286+0000 7f9464ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:36.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.286+0000 7f9464ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:36.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.510+0000 7f946e3b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:36.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.510+0000 7f946e3b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:36.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.510+0000 7f946e3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:36.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.510+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94680627b0 con 0x7f94680a98f0 2024-03-25T19:34:36.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.532+0000 7f9464ff9640 1 -- 172.21.15.134:0/3324391717 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f9458051040 con 0x7f94680a98f0 2024-03-25T19:34:36.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.532+0000 7f9464ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:36.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.532+0000 7f9464ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:36.532 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:36.532 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 10 librados: watch_flush enter 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 10 librados: watch_flush exit 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 10 client.4483.objecter successfully canceled tick 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 >> v1:172.21.15.159:6800/1875874887 conn(0x7f945406ddb0 legacy=0x7f9454070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 10 monclient: shutdown 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 >> v1:172.21.15.159:6789/0 conn(0x7f94680a98f0 legacy=0x7f9468137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 shutdown_connections 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 >> 172.21.15.134:0/3324391717 conn(0x7f94680b7ea0 msgr2=0x7f9468074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 shutdown_connections 2024-03-25T19:34:36.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 1 -- 172.21.15.134:0/3324391717 wait complete. 2024-03-25T19:34:36.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:36.534+0000 7f946e3b6640 1 librados: shutdown 2024-03-25T19:34:36.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:43.354 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:43.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.471+0000 7f735ccf1640 10 monclient: get_monmap_and_config 2024-03-25T19:34:43.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.471+0000 7f735ccf1640 10 monclient: build_initial_monmap 2024-03-25T19:34:43.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.471+0000 7f735ccf1640 10 monclient: monmap: 2024-03-25T19:34:43.471 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:43.471 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:43.471 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:43.472426+0000 2024-03-25T19:34:43.472 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:43.472426+0000 2024-03-25T19:34:43.472 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:43.472 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:43.472 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:43.472 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:43.472 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:43.472 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 1 Processor -- start 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 1 -- start start 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 10 monclient: init 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 10 monclient(hunting): picked mon.noname-a con 0x7f73580b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 10 monclient(hunting): picked mon.noname-b con 0x7f73580b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 10 monclient(hunting): picked mon.noname-c con 0x7f735814be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:43.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.472+0000 7f735ccf1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73580af940 con 0x7f73580b4d60 2024-03-25T19:34:43.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.473+0000 7f735ccf1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73580b0a70 con 0x7f735814be90 2024-03-25T19:34:43.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.473+0000 7f735ccf1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73580b0d50 con 0x7f73580b1730 2024-03-25T19:34:43.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.473+0000 7f735ccf1640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:43.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.473+0000 7f735ccf1640 10 monclient(hunting): authenticate will time out at 6315.631658s 2024-03-25T19:34:43.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.484+0000 7f7356d9d640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f735814be90 0x7f735814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52038/0 (socket says 172.21.15.134:52038) 2024-03-25T19:34:43.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.484+0000 7f7356d9d640 1 -- 172.21.15.134:0/588885085 learned_addr learned my addr 172.21.15.134:0/588885085 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:43.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.499+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517454359 0 0) 0x7f73580b0a70 con 0x7f735814be90 2024-03-25T19:34:43.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.499+0000 7f735559a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.499+0000 7f735559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.499+0000 7f735559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.499+0000 7f735559a640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:34:43.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.499+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7340003520 con 0x7f735814be90 2024-03-25T19:34:43.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391764756 0 0) 0x7f73580af940 con 0x7f73580b4d60 2024-03-25T19:34:43.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:34:43.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73580b0a70 con 0x7f73580b4d60 2024-03-25T19:34:43.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936833453 0 0) 0x7f73580b0d50 con 0x7f73580b1730 2024-03-25T19:34:43.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:34:43.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.501+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73580af940 con 0x7f73580b1730 2024-03-25T19:34:43.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1373970200 0 0) 0x7f7340003520 con 0x7f735814be90 2024-03-25T19:34:43.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 >> v1:172.21.15.134:6790/0 conn(0x7f73580b1730 legacy=0x7f73580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 >> v1:172.21.15.134:6789/0 conn(0x7f73580b4d60 legacy=0x7f735813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 10 monclient: _finish_hunting 0 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 1 monclient: found mon.noname-c 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73580b1030 con 0x7f735814be90 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 10 monclient: _finish_auth 0 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 10 monclient: _check_auth_tickets 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735ccf1640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:34:43.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.506+0000 7f735ccf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.511+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f73500019d0 con 0x7f735814be90 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.511+0000 7f735559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.511+0000 7f735559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.511+0000 7f735559a640 10 monclient: dump: 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:43.511 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:43.512 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:43.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.511+0000 7f735ccf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:43.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7350001c20 con 0x7f735814be90 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735559a640 1 -- 172.21.15.134:0/588885085 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f7350001fb0 con 0x7f735814be90 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735559a640 10 monclient: dump: 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.519 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.520 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:43.520 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:43.520 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:43.520 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:43.520 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:43.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735ccf1640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:43.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.519+0000 7f735ccf1640 4 set_mon_vals no callback set 2024-03-25T19:34:43.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.520+0000 7f735ccf1640 10 monclient: shutdown 2024-03-25T19:34:43.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.520+0000 7f735ccf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:43.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.520+0000 7f735ccf1640 1 -- 172.21.15.134:0/588885085 >> v1:172.21.15.159:6789/0 conn(0x7f735814be90 legacy=0x7f735814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.520+0000 7f735ccf1640 1 -- 172.21.15.134:0/588885085 shutdown_connections 2024-03-25T19:34:43.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.520+0000 7f735ccf1640 1 -- 172.21.15.134:0/588885085 >> 172.21.15.134:0/588885085 conn(0x7f73580617a0 msgr2=0x7f7358061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:43.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.520+0000 7f735ccf1640 1 -- 172.21.15.134:0/588885085 shutdown_connections 2024-03-25T19:34:43.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.520+0000 7f735ccf1640 1 -- 172.21.15.134:0/588885085 wait complete. 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.521+0000 7f735ccf1640 10 monclient: build_initial_monmap 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.521+0000 7f735ccf1640 10 monclient: monmap: 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:43.522 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.522+0000 7f735ccf1640 1 librados: starting msgr at 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.522+0000 7f735ccf1640 1 librados: starting objecter 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.522+0000 7f735ccf1640 1 Processor -- start 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.522+0000 7f735ccf1640 1 -- start start 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.522+0000 7f735ccf1640 1 librados: setting wanted keys 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.522+0000 7f735ccf1640 1 librados: calling monclient init 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.522+0000 7f735ccf1640 10 monclient: init 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 10 monclient(hunting): picked mon.noname-c con 0x7f73580b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 10 monclient(hunting): picked mon.noname-b con 0x7f73580b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 10 monclient(hunting): picked mon.noname-a con 0x7f735814be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73581366f0 con 0x7f735814be90 2024-03-25T19:34:43.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7358137fc0 con 0x7f73580b4d60 2024-03-25T19:34:43.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7358139070 con 0x7f73580b1730 2024-03-25T19:34:43.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:43.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.523+0000 7f735ccf1640 10 monclient(hunting): authenticate will time out at 6315.681891s 2024-03-25T19:34:43.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.528+0000 7f735659c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f73580b1730 0x7f735812fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:54708/0 (socket says 172.21.15.134:54708) 2024-03-25T19:34:43.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.528+0000 7f735659c640 1 -- 172.21.15.134:0/1785034173 learned_addr learned my addr 172.21.15.134:0/1785034173 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:43.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.530+0000 7f7346ffd640 10 client.?.objecter ms_handle_connect 0x7f73580b1730 2024-03-25T19:34:43.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.530+0000 7f7346ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.547+0000 7f7346ffd640 10 client.?.objecter ms_handle_connect 0x7f73580b4d60 2024-03-25T19:34:43.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.547+0000 7f7346ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.548+0000 7f7346ffd640 10 client.?.objecter ms_handle_connect 0x7f735814be90 2024-03-25T19:34:43.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.548+0000 7f7346ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:43.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.555+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872572827 0 0) 0x7f7358137fc0 con 0x7f73580b4d60 2024-03-25T19:34:43.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.555+0000 7f7346ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.555+0000 7f7346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.555+0000 7f7346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.555+0000 7f7346ffd640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:34:43.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.555+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338004eb0 con 0x7f73580b4d60 2024-03-25T19:34:43.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.558+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793723627 0 0) 0x7f7358139070 con 0x7f73580b1730 2024-03-25T19:34:43.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.558+0000 7f7346ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.558+0000 7f7346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.558+0000 7f7346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.558+0000 7f7346ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:34:43.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.558+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7358137fc0 con 0x7f73580b1730 2024-03-25T19:34:43.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.564+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877041344 0 0) 0x7f73581366f0 con 0x7f735814be90 2024-03-25T19:34:43.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.564+0000 7f7346ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:43.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.564+0000 7f7346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:43.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.564+0000 7f7346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:43.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.564+0000 7f7346ffd640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:34:43.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.564+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7358139070 con 0x7f735814be90 2024-03-25T19:34:43.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.572+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 904253659 0 0) 0x7f7338004eb0 con 0x7f73580b4d60 2024-03-25T19:34:43.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.572+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73581366f0 con 0x7f73580b4d60 2024-03-25T19:34:43.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.572+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f734c0018f0 con 0x7f73580b4d60 2024-03-25T19:34:43.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.572+0000 7f7346ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.572+0000 7f7346ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:43.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.572+0000 7f7346ffd640 10 monclient(hunting): dump: 2024-03-25T19:34:43.572 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:43.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.572+0000 7f7346ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:43.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4236402246 0 0) 0x7f73581366f0 con 0x7f73580b4d60 2024-03-25T19:34:43.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 >> v1:172.21.15.134:6790/0 conn(0x7f73580b1730 legacy=0x7f735812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 >> v1:172.21.15.134:6789/0 conn(0x7f735814be90 legacy=0x7f7358144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:43.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:43.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 1 monclient: found mon.b 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7358135f20 con 0x7f73580b4d60 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f7346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 10 monclient: _renew_subs 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7358136110 con 0x7f73580b4d60 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 10 monclient: _renew_subs 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7358130650 con 0x7f73580b4d60 2024-03-25T19:34:43.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.585+0000 7f735ccf1640 1 librados: init done 2024-03-25T19:34:43.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.586+0000 7f735ccf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.586+0000 7f735ccf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.586+0000 7f735ccf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:43.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.586+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7358136450 con 0x7f73580b4d60 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f734c003610 con 0x7f73580b4d60 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7346ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f734c0039a0 con 0x7f73580b4d60 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7346ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7346ffd640 10 monclient: dump: 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.855 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7354d99640 4 set_mon_vals no callback set 2024-03-25T19:34:43.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.855+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f734c003bc0 con 0x7f73580b4d60 2024-03-25T19:34:43.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.856+0000 7f735659c640 10 monclient: get_auth_request con 0x7f733806de00 auth_method 0 2024-03-25T19:34:43.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.857+0000 7f7346ffd640 10 client.4489.objecter ms_handle_connect 0x7f733806de00 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f734c0535a0 con 0x7f73580b4d60 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 10 client.4489.objecter ms_dispatch 0x7f7358000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f734c04f0b0 con 0x7f73580b4d60 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:43.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:43.861+0000 7f7346ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:44.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.106+0000 7f735ccf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:44.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.106+0000 7f735ccf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:44.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.106+0000 7f735ccf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:44.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.106+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73581362a0 con 0x7f73580b4d60 2024-03-25T19:34:44.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.139+0000 7f7346ffd640 1 -- 172.21.15.134:0/1785034173 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f734c050fe0 con 0x7f73580b4d60 2024-03-25T19:34:44.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.139+0000 7f7346ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:44.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.139+0000 7f7346ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:44.139 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:44.139 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:44.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.141+0000 7f735ccf1640 10 librados: watch_flush enter 2024-03-25T19:34:44.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.141+0000 7f735ccf1640 10 librados: watch_flush exit 2024-03-25T19:34:44.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T19:34:44.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 10 client.4489.objecter successfully canceled tick 2024-03-25T19:34:44.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 >> v1:172.21.15.159:6800/1875874887 conn(0x7f733806de00 legacy=0x7f73380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 10 monclient: shutdown 2024-03-25T19:34:44.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:44.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 >> v1:172.21.15.159:6789/0 conn(0x7f73580b4d60 legacy=0x7f7358143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:44.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 shutdown_connections 2024-03-25T19:34:44.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 >> 172.21.15.134:0/1785034173 conn(0x7f73580617a0 msgr2=0x7f735813c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:44.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 shutdown_connections 2024-03-25T19:34:44.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 1 -- 172.21.15.134:0/1785034173 wait complete. 2024-03-25T19:34:44.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:44.142+0000 7f735ccf1640 1 librados: shutdown 2024-03-25T19:34:44.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:45.699 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:50.371 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.710 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:50.382 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.737 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:50.410 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.767 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:50.440 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.772 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:50.445 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.774 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:50.421 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.777 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:50.424 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.796 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:50.469 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:45.808 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:50.455 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:50.961 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:51.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.081+0000 7fa3fa932640 10 monclient: get_monmap_and_config 2024-03-25T19:34:51.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.081+0000 7fa3fa932640 10 monclient: build_initial_monmap 2024-03-25T19:34:51.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.081+0000 7fa3fa932640 10 monclient: monmap: 2024-03-25T19:34:51.081 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:51.082526+0000 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:51.082526+0000 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:51.082 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.082+0000 7fa3fa932640 1 Processor -- start 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.082+0000 7fa3fa932640 1 -- start start 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.082+0000 7fa3fa932640 10 monclient: init 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f41430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f4142520 con 0x7fa3f40a98f0 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f40b28a0 con 0x7fa3f413f710 2024-03-25T19:34:51.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f40b2a30 con 0x7fa3f41430c0 2024-03-25T19:34:51.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:51.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.083+0000 7fa3fa932640 10 monclient(hunting): authenticate will time out at 6323.242116s 2024-03-25T19:34:51.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.093+0000 7fa3f3fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa3f40a98f0 0x7fa3f40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:54376/0 (socket says 172.21.15.134:54376) 2024-03-25T19:34:51.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.093+0000 7fa3f3fff640 1 -- 172.21.15.134:0/3840843715 learned_addr learned my addr 172.21.15.134:0/3840843715 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:51.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.101+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276108944 0 0) 0x7fa3f40b2a30 con 0x7fa3f41430c0 2024-03-25T19:34:51.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.101+0000 7fa3f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:51.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.101+0000 7fa3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:51.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.101+0000 7fa3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:51.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.101+0000 7fa3f2ffd640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:34:51.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.101+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc003520 con 0x7fa3f41430c0 2024-03-25T19:34:51.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.108+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572061645 0 0) 0x7fa3f40b28a0 con 0x7fa3f413f710 2024-03-25T19:34:51.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.108+0000 7fa3f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:51.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.108+0000 7fa3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:51.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.108+0000 7fa3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:51.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.108+0000 7fa3f2ffd640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:34:51.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.108+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f40b2a30 con 0x7fa3f413f710 2024-03-25T19:34:51.111 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.111+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166359183 0 0) 0x7fa3f4142520 con 0x7fa3f40a98f0 2024-03-25T19:34:51.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.111+0000 7fa3f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:51.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.111+0000 7fa3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:51.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.111+0000 7fa3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:51.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.112+0000 7fa3f2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:34:51.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.112+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f40b28a0 con 0x7fa3f40a98f0 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2482419747 0 0) 0x7fa3f40b2a30 con 0x7fa3f413f710 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 >> v1:172.21.15.134:6790/0 conn(0x7fa3f41430c0 legacy=0x7fa3f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 >> v1:172.21.15.134:6789/0 conn(0x7fa3f40a98f0 legacy=0x7fa3f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f40b44c0 con 0x7fa3f413f710 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa3e8001670 con 0x7fa3f413f710 2024-03-25T19:34:51.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3fa932640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3f2ffd640 10 monclient: dump: 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:51.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.116+0000 7fa3fa932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:51.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3e80017d0 con 0x7fa3f413f710 2024-03-25T19:34:51.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:51.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3f2ffd640 1 -- 172.21.15.134:0/3840843715 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa3e8001be0 con 0x7fa3f413f710 2024-03-25T19:34:51.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3fa932640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3fa932640 4 set_mon_vals no callback set 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.133+0000 7fa3f2ffd640 10 monclient: dump: 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:51.134 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.134+0000 7fa3fa932640 10 monclient: shutdown 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.134+0000 7fa3fa932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.134+0000 7fa3fa932640 1 -- 172.21.15.134:0/3840843715 >> v1:172.21.15.159:6789/0 conn(0x7fa3f413f710 legacy=0x7fa3f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.134+0000 7fa3fa932640 1 -- 172.21.15.134:0/3840843715 shutdown_connections 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.134+0000 7fa3fa932640 1 -- 172.21.15.134:0/3840843715 >> 172.21.15.134:0/3840843715 conn(0x7fa3f40b7ea0 msgr2=0x7fa3f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.134+0000 7fa3fa932640 1 -- 172.21.15.134:0/3840843715 shutdown_connections 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.134+0000 7fa3fa932640 1 -- 172.21.15.134:0/3840843715 wait complete. 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.135+0000 7fa3fa932640 10 monclient: build_initial_monmap 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.135+0000 7fa3fa932640 10 monclient: monmap: 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:51.136 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.135+0000 7fa3fa932640 1 librados: starting msgr at 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.135+0000 7fa3fa932640 1 librados: starting objecter 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.135+0000 7fa3fa932640 1 Processor -- start 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 1 -- start start 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 1 librados: setting wanted keys 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 1 librados: calling monclient init 2024-03-25T19:34:51.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient: init 2024-03-25T19:34:51.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f41430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f4136510 con 0x7fa3f40a98f0 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f40abb00 con 0x7fa3f413f710 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f41da990 con 0x7fa3f41430c0 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.136+0000 7fa3fa932640 10 monclient(hunting): authenticate will time out at 6323.295517s 2024-03-25T19:34:51.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.139+0000 7fa3f37fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa3f413f710 0x7fa3f41383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:57152/0 (socket says 172.21.15.134:57152) 2024-03-25T19:34:51.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.139+0000 7fa3f37fe640 1 -- 172.21.15.134:0/3235242980 learned_addr learned my addr 172.21.15.134:0/3235242980 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:51.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.142+0000 7fa3f0ff9640 10 client.?.objecter ms_handle_connect 0x7fa3f413f710 2024-03-25T19:34:51.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.142+0000 7fa3f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:51.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.146+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474734493 0 0) 0x7fa3f40abb00 con 0x7fa3f413f710 2024-03-25T19:34:51.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.146+0000 7fa3f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:51.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.146+0000 7fa3f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:51.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.146+0000 7fa3f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:51.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.146+0000 7fa3f0ff9640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:34:51.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.146+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e0004eb0 con 0x7fa3f413f710 2024-03-25T19:34:51.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.148+0000 7fa3f0ff9640 10 client.?.objecter ms_handle_connect 0x7fa3f40a98f0 2024-03-25T19:34:51.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.148+0000 7fa3f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:51.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.149+0000 7fa3f0ff9640 10 client.?.objecter ms_handle_connect 0x7fa3f41430c0 2024-03-25T19:34:51.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.149+0000 7fa3f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:51.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.154+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206916546 0 0) 0x7fa3f41da990 con 0x7fa3f41430c0 2024-03-25T19:34:51.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.154+0000 7fa3f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:51.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.154+0000 7fa3f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:51.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.154+0000 7fa3f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:51.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.154+0000 7fa3f0ff9640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:34:51.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.154+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f40abb00 con 0x7fa3f41430c0 2024-03-25T19:34:51.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.156+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265144901 0 0) 0x7fa3e0004eb0 con 0x7fa3f413f710 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.156+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f41da990 con 0x7fa3f413f710 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.156+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa3e8001670 con 0x7fa3f413f710 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.156+0000 7fa3f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.156+0000 7fa3f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.156+0000 7fa3f0ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:51.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.156+0000 7fa3f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:51.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.157+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316605826 0 0) 0x7fa3f4136510 con 0x7fa3f40a98f0 2024-03-25T19:34:51.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.157+0000 7fa3f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:51.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.157+0000 7fa3f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:51.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.157+0000 7fa3f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:51.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.157+0000 7fa3f0ff9640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:34:51.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.157+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e0004eb0 con 0x7fa3f40a98f0 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 915462077 0 0) 0x7fa3f41da990 con 0x7fa3f413f710 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 >> v1:172.21.15.134:6790/0 conn(0x7fa3f41430c0 legacy=0x7fa3f41e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 >> v1:172.21.15.134:6789/0 conn(0x7fa3f40a98f0 legacy=0x7fa3f41daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 1 monclient: found mon.b 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4137820 con 0x7fa3f413f710 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 10 monclient: _renew_subs 2024-03-25T19:34:51.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f41db5c0 con 0x7fa3f413f710 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 10 monclient: _renew_subs 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f4138c90 con 0x7fa3f413f710 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.165+0000 7fa3fa932640 1 librados: init done 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.166+0000 7fa3fa932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.166+0000 7fa3fa932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:51.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.166+0000 7fa3fa932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:51.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.166+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f40b3ef0 con 0x7fa3f413f710 2024-03-25T19:34:51.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.192+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3e80038a0 con 0x7fa3f413f710 2024-03-25T19:34:51.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.192+0000 7fa3f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:51.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.192+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa3e8003cf0 con 0x7fa3f413f710 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.192+0000 7fa3f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.192+0000 7fa3f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.192+0000 7fa3f0ff9640 10 monclient: dump: 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:51.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.192+0000 7fa3f27fc640 4 set_mon_vals no callback set 2024-03-25T19:34:51.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.193+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fa3e8003f10 con 0x7fa3f413f710 2024-03-25T19:34:51.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.194+0000 7fa3f3fff640 10 monclient: get_auth_request con 0x7fa3e006dc10 auth_method 0 2024-03-25T19:34:51.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.194+0000 7fa3f0ff9640 10 client.4495.objecter ms_handle_connect 0x7fa3e006dc10 2024-03-25T19:34:51.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.457+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fa3e8054940 con 0x7fa3f413f710 2024-03-25T19:34:51.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.457+0000 7fa3f0ff9640 10 client.4495.objecter ms_dispatch 0x7fa3f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:51.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.457+0000 7fa3f0ff9640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:51.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.457+0000 7fa3f0ff9640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:51.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.457+0000 7fa3f0ff9640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T19:34:51.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.459+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3e8050450 con 0x7fa3f413f710 2024-03-25T19:34:51.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.459+0000 7fa3f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:51.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.459+0000 7fa3f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:51.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.685+0000 7fa3fa932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:51.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.685+0000 7fa3fa932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:51.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.685+0000 7fa3fa932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:34:51.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.685+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3f414f320 con 0x7fa3f413f710 2024-03-25T19:34:51.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.706+0000 7fa3f0ff9640 1 -- 172.21.15.134:0/3235242980 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fa3e8052380 con 0x7fa3f413f710 2024-03-25T19:34:51.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.706+0000 7fa3f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:51.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.706+0000 7fa3f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:51.707 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:51.707 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:51.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 10 librados: watch_flush enter 2024-03-25T19:34:51.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 10 librados: watch_flush exit 2024-03-25T19:34:51.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T19:34:51.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 10 client.4495.objecter successfully canceled tick 2024-03-25T19:34:51.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 >> v1:172.21.15.159:6800/1875874887 conn(0x7fa3e006dc10 legacy=0x7fa3e00700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:51.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 10 monclient: shutdown 2024-03-25T19:34:51.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:51.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 >> v1:172.21.15.159:6789/0 conn(0x7fa3f413f710 legacy=0x7fa3f41383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:51.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 shutdown_connections 2024-03-25T19:34:51.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 >> 172.21.15.134:0/3235242980 conn(0x7fa3f40b7ea0 msgr2=0x7fa3f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:51.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 shutdown_connections 2024-03-25T19:34:51.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 1 -- 172.21.15.134:0/3235242980 wait complete. 2024-03-25T19:34:51.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:51.709+0000 7fa3fa932640 1 librados: shutdown 2024-03-25T19:34:51.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:58.528 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:58.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.639+0000 7f8653504640 10 monclient: get_monmap_and_config 2024-03-25T19:34:58.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.639+0000 7f8653504640 10 monclient: build_initial_monmap 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.639+0000 7f8653504640 10 monclient: monmap: 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:34:58.640873+0000 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:34:58.640873+0000 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:34:58.640 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:58.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.640+0000 7f8653504640 1 Processor -- start 2024-03-25T19:34:58.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.640+0000 7f8653504640 1 -- start start 2024-03-25T19:34:58.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.640+0000 7f8653504640 10 monclient: init 2024-03-25T19:34:58.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:58.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:58.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 10 monclient(hunting): picked mon.noname-b con 0x7f864c0705b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 10 monclient(hunting): picked mon.noname-c con 0x7f864c1402d0 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 10 monclient(hunting): picked mon.noname-a con 0x7f864c149ae0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c0718c0 con 0x7f864c149ae0 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c0714c0 con 0x7f864c1402d0 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c072b50 con 0x7f864c0705b0 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.641+0000 7f8653504640 10 monclient(hunting): authenticate will time out at 6330.800328s 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.642+0000 7f8650a9f640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f864c1402d0 0x7f864c140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:33586/0 (socket says 172.21.15.134:33586) 2024-03-25T19:34:58.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.642+0000 7f8650a9f640 1 -- 172.21.15.134:0/332762503 learned_addr learned my addr 172.21.15.134:0/332762503 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:58.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.648+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371243633 0 0) 0x7f864c0714c0 con 0x7f864c1402d0 2024-03-25T19:34:58.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.648+0000 7f863ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:58.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.648+0000 7f863ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:58.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.648+0000 7f863ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:58.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.648+0000 7f863ffff640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:34:58.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.648+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8638003520 con 0x7f864c1402d0 2024-03-25T19:34:58.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.658+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408080071 0 0) 0x7f864c072b50 con 0x7f864c0705b0 2024-03-25T19:34:58.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.658+0000 7f863ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:58.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.658+0000 7f863ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:58.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.658+0000 7f863ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:58.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.658+0000 7f863ffff640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:34:58.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.658+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c0714c0 con 0x7f864c0705b0 2024-03-25T19:34:58.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641637251 0 0) 0x7f8638003520 con 0x7f864c1402d0 2024-03-25T19:34:58.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 >> v1:172.21.15.134:6790/0 conn(0x7f864c0705b0 legacy=0x7f864c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:58.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 >> v1:172.21.15.134:6789/0 conn(0x7f864c149ae0 legacy=0x7f864c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 1 monclient: found mon.noname-c 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c13dd10 con 0x7f864c1402d0 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 10 monclient: _finish_auth 0 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8648001690 con 0x7f864c1402d0 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f8653504640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f863ffff640 10 monclient: dump: 2024-03-25T19:34:58.665 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:58.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.664+0000 7f8653504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:58.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86480017f0 con 0x7f864c1402d0 2024-03-25T19:34:58.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f863ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f863ffff640 1 -- 172.21.15.134:0/332762503 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8648001c00 con 0x7f864c1402d0 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f8653504640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f8653504640 4 set_mon_vals no callback set 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f863ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f863ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.686+0000 7f863ffff640 10 monclient: dump: 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:58.687 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:58.688 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:58.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.687+0000 7f8653504640 10 monclient: shutdown 2024-03-25T19:34:58.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.687+0000 7f8653504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:58.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.687+0000 7f8653504640 1 -- 172.21.15.134:0/332762503 >> v1:172.21.15.159:6789/0 conn(0x7f864c1402d0 legacy=0x7f864c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:58.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.687+0000 7f8653504640 1 -- 172.21.15.134:0/332762503 shutdown_connections 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.687+0000 7f8653504640 1 -- 172.21.15.134:0/332762503 >> 172.21.15.134:0/332762503 conn(0x7f864c0b5d10 msgr2=0x7f864c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.687+0000 7f8653504640 1 -- 172.21.15.134:0/332762503 shutdown_connections 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.687+0000 7f8653504640 1 -- 172.21.15.134:0/332762503 wait complete. 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.688+0000 7f8653504640 10 monclient: build_initial_monmap 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.688+0000 7f8653504640 10 monclient: monmap: 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 librados: starting msgr at 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 librados: starting objecter 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 Processor -- start 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 -- start start 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 librados: setting wanted keys 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 librados: calling monclient init 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient: init 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient(hunting): picked mon.noname-b con 0x7f864c0705b0 addr v1:172.21.15.159:6789/0 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient(hunting): picked mon.noname-c con 0x7f864c1402d0 addr v1:172.21.15.134:6790/0 2024-03-25T19:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient(hunting): picked mon.noname-a con 0x7f864c149ae0 addr v1:172.21.15.134:6789/0 2024-03-25T19:34:58.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c1d58d0 con 0x7f864c149ae0 2024-03-25T19:34:58.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c144180 con 0x7f864c0705b0 2024-03-25T19:34:58.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c142f80 con 0x7f864c1402d0 2024-03-25T19:34:58.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:58.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.689+0000 7f8653504640 10 monclient(hunting): authenticate will time out at 6330.848618s 2024-03-25T19:34:58.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.695+0000 7f8650a9f640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f864c1402d0 0x7f864c1d5d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:37358/0 (socket says 172.21.15.134:37358) 2024-03-25T19:34:58.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.695+0000 7f8650a9f640 1 -- 172.21.15.134:0/1584171829 learned_addr learned my addr 172.21.15.134:0/1584171829 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:34:58.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.697+0000 7f863dffb640 10 client.?.objecter ms_handle_connect 0x7f864c1402d0 2024-03-25T19:34:58.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.697+0000 7f863dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:58.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.699+0000 7f863dffb640 10 client.?.objecter ms_handle_connect 0x7f864c0705b0 2024-03-25T19:34:58.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.699+0000 7f863dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:58.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.702+0000 7f863dffb640 10 client.?.objecter ms_handle_connect 0x7f864c149ae0 2024-03-25T19:34:58.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.702+0000 7f863dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:58.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.707+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199223177 0 0) 0x7f864c142f80 con 0x7f864c1402d0 2024-03-25T19:34:58.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.707+0000 7f863dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:58.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.707+0000 7f863dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:58.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.707+0000 7f863dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:58.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.707+0000 7f863dffb640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:34:58.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.707+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630004eb0 con 0x7f864c1402d0 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27841557 0 0) 0x7f864c144180 con 0x7f864c0705b0 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c142f80 con 0x7f864c0705b0 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714086440 0 0) 0x7f864c1d58d0 con 0x7f864c149ae0 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:34:58.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.710+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c144180 con 0x7f864c149ae0 2024-03-25T19:34:58.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.719+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3920748016 0 0) 0x7f864c144180 con 0x7f864c149ae0 2024-03-25T19:34:58.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.719+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f864c1d58d0 con 0x7f864c149ae0 2024-03-25T19:34:58.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.719+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8644001a90 con 0x7f864c149ae0 2024-03-25T19:34:58.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.719+0000 7f863dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.719+0000 7f863dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.719+0000 7f863dffb640 10 monclient(hunting): dump: 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:58.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.719+0000 7f863dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.724+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111202526 0 0) 0x7f8630004eb0 con 0x7f864c1402d0 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.724+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f864c144180 con 0x7f864c1402d0 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.724+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8648001bc0 con 0x7f864c1402d0 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.724+0000 7f863dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.724+0000 7f863dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.724+0000 7f863dffb640 10 monclient(hunting): dump: 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.724 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.725 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:58.725 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:58.725 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:58.725 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:58.725 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:58.725 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.730+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2471059551 0 0) 0x7f864c142f80 con 0x7f864c0705b0 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.730+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8630004eb0 con 0x7f864c0705b0 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.730+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8640001930 con 0x7f864c0705b0 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.730+0000 7f863dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.730+0000 7f863dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.730+0000 7f863dffb640 10 monclient(hunting): dump: 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:58.730 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:58.731 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:58.731 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:58.731 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:58.731 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:58.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3052789111 0 0) 0x7f864c144180 con 0x7f864c1402d0 2024-03-25T19:34:58.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 >> v1:172.21.15.159:6789/0 conn(0x7f864c0705b0 legacy=0x7f864c142870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:58.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 >> v1:172.21.15.134:6789/0 conn(0x7f864c149ae0 legacy=0x7f864c148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:58.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 1 monclient: found mon.c 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c141d90 con 0x7f864c1402d0 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f863dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 5 monclient: authenticate success, global_id 4508 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 10 monclient: _renew_subs 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c1d6470 con 0x7f864c1402d0 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 10 monclient: _renew_subs 2024-03-25T19:34:58.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:58.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c1433a0 con 0x7f864c1402d0 2024-03-25T19:34:58.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.735+0000 7f8653504640 1 librados: init done 2024-03-25T19:34:58.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.736+0000 7f8653504640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:58.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.736+0000 7f8653504640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:58.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.736+0000 7f8653504640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:58.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:58.736+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c0627b0 con 0x7f864c1402d0 2024-03-25T19:34:59.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.031+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86480011f0 con 0x7f864c1402d0 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.031+0000 7f863dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.031+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8648004cb0 con 0x7f864c1402d0 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.031+0000 7f863dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.031+0000 7f863dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.031+0000 7f863dffb640 10 monclient: dump: 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:34:59.032 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:34:59.033 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:34:59.033 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:34:59.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.031+0000 7f863f7fe640 4 set_mon_vals no callback set 2024-03-25T19:34:59.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.032+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f864801acc0 con 0x7f864c1402d0 2024-03-25T19:34:59.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.033+0000 7f86512a0640 10 monclient: get_auth_request con 0x7f863006e080 auth_method 0 2024-03-25T19:34:59.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.033+0000 7f863dffb640 10 client.4508.objecter ms_handle_connect 0x7f863006e080 2024-03-25T19:34:59.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.037+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f86480549a0 con 0x7f864c1402d0 2024-03-25T19:34:59.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.037+0000 7f863dffb640 10 client.4508.objecter ms_dispatch 0x7f864c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:59.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.037+0000 7f863dffb640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:59.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.037+0000 7f863dffb640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:59.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.038+0000 7f863dffb640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-25T19:34:59.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.038+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86480502e0 con 0x7f864c1402d0 2024-03-25T19:34:59.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.038+0000 7f863dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:59.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.038+0000 7f863dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:59.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.286+0000 7f8653504640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:59.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.286+0000 7f8653504640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:59.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.286+0000 7f8653504640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:34:59.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.286+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f864c1d5a80 con 0x7f864c1402d0 2024-03-25T19:34:59.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.320+0000 7f863dffb640 1 -- 172.21.15.134:0/1584171829 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f864c1d5a80 con 0x7f864c1402d0 2024-03-25T19:34:59.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.320+0000 7f863dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:59.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.320+0000 7f863dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:59.321 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:34:59.321 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:59.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 10 librados: watch_flush enter 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 10 librados: watch_flush exit 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 10 client.4508.objecter successfully canceled tick 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 >> v1:172.21.15.159:6800/1875874887 conn(0x7f863006e080 legacy=0x7f8630070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 10 monclient: shutdown 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 >> v1:172.21.15.134:6790/0 conn(0x7f864c1402d0 legacy=0x7f864c1d5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 shutdown_connections 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 >> 172.21.15.134:0/1584171829 conn(0x7f864c0b5d10 msgr2=0x7f864c140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:59.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 shutdown_connections 2024-03-25T19:34:59.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.323+0000 7f8653504640 1 -- 172.21.15.134:0/1584171829 wait complete. 2024-03-25T19:34:59.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:34:59.324+0000 7f8653504640 1 librados: shutdown 2024-03-25T19:34:59.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:05.741 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:10.414 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.745 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:10.418 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.774 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:10.446 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.809 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:10.456 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.812 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:10.459 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.820 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:10.493 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.825 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:10.497 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.847 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:10.494 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:05.861 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:10.533 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:06.143 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.258+0000 7f74b5c81640 10 monclient: get_monmap_and_config 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.258+0000 7f74b5c81640 10 monclient: build_initial_monmap 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.258+0000 7f74b5c81640 10 monclient: monmap: 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:06.259356+0000 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:06.259356+0000 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:06.258 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:06.259 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:06.259 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:06.259 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:06.259 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:06.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.259+0000 7f74b5c81640 1 Processor -- start 2024-03-25T19:35:06.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.259+0000 7f74b5c81640 1 -- start start 2024-03-25T19:35:06.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.259+0000 7f74b5c81640 10 monclient: init 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 10 monclient(hunting): picked mon.noname-c con 0x7f74b00b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 10 monclient(hunting): picked mon.noname-a con 0x7f74b00b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 10 monclient(hunting): picked mon.noname-b con 0x7f74b014be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00af940 con 0x7f74b00b1730 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00b0a70 con 0x7f74b00b4d60 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00b0d50 con 0x7f74b014be90 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:06.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.260+0000 7f74b5c81640 10 monclient(hunting): authenticate will time out at 6338.419509s 2024-03-25T19:35:06.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.274+0000 7f74aeffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f74b00b1730 0x7f74b00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:44838/0 (socket says 172.21.15.134:44838) 2024-03-25T19:35:06.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.274+0000 7f74aeffd640 1 -- 172.21.15.134:0/3503556004 learned_addr learned my addr 172.21.15.134:0/3503556004 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:06.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933374060 0 0) 0x7f74b00b0d50 con 0x7f74b014be90 2024-03-25T19:35:06.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:06.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:06.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:06.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:35:06.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7498003520 con 0x7f74b014be90 2024-03-25T19:35:06.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539652351 0 0) 0x7f74b00af940 con 0x7f74b00b1730 2024-03-25T19:35:06.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:06.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:06.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:06.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:35:06.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.288+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b00b0d50 con 0x7f74b00b1730 2024-03-25T19:35:06.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.299+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651335020 0 0) 0x7f74b00b0a70 con 0x7f74b00b4d60 2024-03-25T19:35:06.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.299+0000 7f74ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:06.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.299+0000 7f74ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:06.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.299+0000 7f74ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:06.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.299+0000 7f74ae7fc640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:35:06.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.299+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b00af940 con 0x7f74b00b4d60 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345950512 0 0) 0x7f74b00b0d50 con 0x7f74b00b1730 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 >> v1:172.21.15.134:6790/0 conn(0x7f74b014be90 legacy=0x7f74b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 >> v1:172.21.15.159:6789/0 conn(0x7f74b00b4d60 legacy=0x7f74b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 1 monclient: found mon.noname-a 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00b1030 con 0x7f74b00b1730 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:06.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f74a4001670 con 0x7f74b00b1730 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74ae7fc640 10 monclient: dump: 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:06.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74b5c81640 5 monclient: authenticate success, global_id 4545 2024-03-25T19:35:06.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.310+0000 7f74b5c81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74a40017d0 con 0x7f74b00b1730 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74ae7fc640 1 -- 172.21.15.134:0/3503556004 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f74a4001be0 con 0x7f74b00b1730 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74b5c81640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74b5c81640 4 set_mon_vals no callback set 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74ae7fc640 10 monclient: dump: 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.322 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.323 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:06.323 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:06.323 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:06.323 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:06.323 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:06.323 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74b5c81640 10 monclient: shutdown 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.322+0000 7f74b5c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.323+0000 7f74b5c81640 1 -- 172.21.15.134:0/3503556004 >> v1:172.21.15.134:6789/0 conn(0x7f74b00b1730 legacy=0x7f74b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.323+0000 7f74b5c81640 1 -- 172.21.15.134:0/3503556004 shutdown_connections 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.323+0000 7f74b5c81640 1 -- 172.21.15.134:0/3503556004 >> 172.21.15.134:0/3503556004 conn(0x7f74b00617a0 msgr2=0x7f74b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.323+0000 7f74b5c81640 1 -- 172.21.15.134:0/3503556004 shutdown_connections 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.323+0000 7f74b5c81640 1 -- 172.21.15.134:0/3503556004 wait complete. 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.323+0000 7f74b5c81640 10 monclient: build_initial_monmap 2024-03-25T19:35:06.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.323+0000 7f74b5c81640 10 monclient: monmap: 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 1 librados: starting msgr at 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 1 librados: starting objecter 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 1 Processor -- start 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 1 -- start start 2024-03-25T19:35:06.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 1 librados: setting wanted keys 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 1 librados: calling monclient init 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 10 monclient: init 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.324+0000 7f74b5c81640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 10 monclient(hunting): picked mon.noname-a con 0x7f74b00b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 10 monclient(hunting): picked mon.noname-b con 0x7f74b00b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 10 monclient(hunting): picked mon.noname-c con 0x7f74b014be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b013d0f0 con 0x7f74b00b1730 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b0140380 con 0x7f74b00b4d60 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b0141820 con 0x7f74b014be90 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:06.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.325+0000 7f74b5c81640 10 monclient(hunting): authenticate will time out at 6338.483829s 2024-03-25T19:35:06.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.328+0000 7f74aeffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f74b00b4d60 0x7f74b0132d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:45000/0 (socket says 172.21.15.134:45000) 2024-03-25T19:35:06.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.328+0000 7f74aeffd640 1 -- 172.21.15.134:0/2960070713 learned_addr learned my addr 172.21.15.134:0/2960070713 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:06.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.329+0000 7f7497fff640 10 client.?.objecter ms_handle_connect 0x7f74b00b4d60 2024-03-25T19:35:06.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.329+0000 7f7497fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:06.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.335+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542562793 0 0) 0x7f74b0140380 con 0x7f74b00b4d60 2024-03-25T19:35:06.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.335+0000 7f7497fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:06.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.335+0000 7f7497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:06.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.335+0000 7f7497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:06.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.335+0000 7f7497fff640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:35:06.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.335+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f749c004eb0 con 0x7f74b00b4d60 2024-03-25T19:35:06.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.341+0000 7f7497fff640 10 client.?.objecter ms_handle_connect 0x7f74b014be90 2024-03-25T19:35:06.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.341+0000 7f7497fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:06.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.345+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2165323026 0 0) 0x7f749c004eb0 con 0x7f74b00b4d60 2024-03-25T19:35:06.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.345+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b0140380 con 0x7f74b00b4d60 2024-03-25T19:35:06.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.345+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f74a40047e0 con 0x7f74b00b4d60 2024-03-25T19:35:06.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.345+0000 7f7497fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.346+0000 7f7497fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.346+0000 7f7497fff640 10 monclient(hunting): dump: 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:06.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.346+0000 7f7497fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:06.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.348+0000 7f7497fff640 10 client.?.objecter ms_handle_connect 0x7f74b00b1730 2024-03-25T19:35:06.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.348+0000 7f7497fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:06.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.352+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861220844 0 0) 0x7f74b0141820 con 0x7f74b014be90 2024-03-25T19:35:06.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.352+0000 7f7497fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:06.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.352+0000 7f7497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:06.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.352+0000 7f7497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:06.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.352+0000 7f7497fff640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:35:06.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.352+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f749c004eb0 con 0x7f74b014be90 2024-03-25T19:35:06.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2802697615 0 0) 0x7f74b0140380 con 0x7f74b00b4d60 2024-03-25T19:35:06.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 >> v1:172.21.15.134:6790/0 conn(0x7f74b014be90 legacy=0x7f74b01dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:06.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 >> v1:172.21.15.134:6789/0 conn(0x7f74b00b1730 legacy=0x7f74b0141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 1 monclient: found mon.b 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b01419b0 con 0x7f74b00b4d60 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f7497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 10 monclient: _renew_subs 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b0143140 con 0x7f74b00b4d60 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:06.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 10 monclient: _renew_subs 2024-03-25T19:35:06.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:06.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b0133680 con 0x7f74b00b4d60 2024-03-25T19:35:06.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.360+0000 7f74b5c81640 1 librados: init done 2024-03-25T19:35:06.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.361+0000 7f74b5c81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:06.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.361+0000 7f74b5c81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:06.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.361+0000 7f74b5c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:06.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.361+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b0141b40 con 0x7f74b00b4d60 2024-03-25T19:35:06.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.645+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74a4004bf0 con 0x7f74b00b4d60 2024-03-25T19:35:06.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.645+0000 7f7497fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:06.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.645+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f74a4004f00 con 0x7f74b00b4d60 2024-03-25T19:35:06.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.645+0000 7f7497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:06.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.645+0000 7f7497fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:06.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.645+0000 7f7497fff640 10 monclient: dump: 2024-03-25T19:35:06.645 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:06.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.645+0000 7f74adffb640 4 set_mon_vals no callback set 2024-03-25T19:35:06.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.646+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f74a401aed0 con 0x7f74b00b4d60 2024-03-25T19:35:06.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.647+0000 7f74af7fe640 10 monclient: get_auth_request con 0x7f749c06dd40 auth_method 0 2024-03-25T19:35:06.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.647+0000 7f7497fff640 10 client.4507.objecter ms_handle_connect 0x7f749c06dd40 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f74a4054a80 con 0x7f74b00b4d60 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 10 client.4507.objecter ms_dispatch 0x7f74b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74a40503c0 con 0x7f74b00b4d60 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:06.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.652+0000 7f7497fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:06.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.869+0000 7f74b5c81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:06.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.869+0000 7f74b5c81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:06.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.869+0000 7f74b5c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:06.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.869+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b01432d0 con 0x7f74b00b4d60 2024-03-25T19:35:06.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.941+0000 7f7497fff640 1 -- 172.21.15.134:0/2960070713 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f74a40524c0 con 0x7f74b00b4d60 2024-03-25T19:35:06.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.941+0000 7f7497fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:06.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.941+0000 7f7497fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:06.942 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:35:06.942 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:06.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 10 librados: watch_flush enter 2024-03-25T19:35:06.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 10 librados: watch_flush exit 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 10 client.4507.objecter successfully canceled tick 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 >> v1:172.21.15.159:6800/1875874887 conn(0x7f749c06dd40 legacy=0x7f749c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 10 monclient: shutdown 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 >> v1:172.21.15.159:6789/0 conn(0x7f74b00b4d60 legacy=0x7f74b0132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 shutdown_connections 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 >> 172.21.15.134:0/2960070713 conn(0x7f74b00617a0 msgr2=0x7f74b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 shutdown_connections 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 1 -- 172.21.15.134:0/2960070713 wait complete. 2024-03-25T19:35:06.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:06.944+0000 7f74b5c81640 1 librados: shutdown 2024-03-25T19:35:06.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:13.764 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:13.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.931+0000 7f6060f8f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.931+0000 7f6060f8f640 10 monclient: build_initial_monmap 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.931+0000 7f6060f8f640 10 monclient: monmap: 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:13.932836+0000 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:13.932836+0000 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:13.932 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:13.933 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:13.933 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:13.933 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.932+0000 7f6060f8f640 1 Processor -- start 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.932+0000 7f6060f8f640 1 -- start start 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.932+0000 7f6060f8f640 10 monclient: init 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 10 monclient(hunting): picked mon.noname-c con 0x7f605c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 10 monclient(hunting): picked mon.noname-a con 0x7f605c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:13.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 10 monclient(hunting): picked mon.noname-b con 0x7f605c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:13.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c142520 con 0x7f605c13f710 2024-03-25T19:35:13.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c0b28a0 con 0x7f605c0a98f0 2024-03-25T19:35:13.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c0b2a30 con 0x7f605c1430c0 2024-03-25T19:35:13.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:13.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.933+0000 7f6060f8f640 10 monclient(hunting): authenticate will time out at 6346.091969s 2024-03-25T19:35:13.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.947+0000 7f6059d9b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f605c13f710 0x7f605c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:57266/0 (socket says 172.21.15.134:57266) 2024-03-25T19:35:13.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.947+0000 7f6059d9b640 1 -- 172.21.15.134:0/4086464341 learned_addr learned my addr 172.21.15.134:0/4086464341 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:13.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.950+0000 7f605ad9d640 1 -- 172.21.15.134:0/4086464341 >> v1:172.21.15.134:6790/0 conn(0x7f605c1430c0 legacy=0x7f605c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T19:35:13.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.950+0000 7f605ad9d640 1 -- 172.21.15.134:0/4086464341 >> v1:172.21.15.134:6790/0 conn(0x7f605c1430c0 legacy=0x7f605c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:35:13.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.950+0000 7f605ad9d640 1 --1- 172.21.15.134:0/4086464341 >> v1:172.21.15.134:6790/0 conn(0x7f605c1430c0 0x7f605c14dc30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:35:13.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.958+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406333905 0 0) 0x7f605c0b28a0 con 0x7f605c0a98f0 2024-03-25T19:35:13.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.958+0000 7f605959a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:13.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.958+0000 7f605959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:13.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.958+0000 7f605959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:13.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.958+0000 7f605959a640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:35:13.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.958+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6044003520 con 0x7f605c0a98f0 2024-03-25T19:35:13.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.962+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182484085 0 0) 0x7f605c142520 con 0x7f605c13f710 2024-03-25T19:35:13.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.962+0000 7f605959a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:13.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.962+0000 7f605959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:13.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.962+0000 7f605959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:13.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.962+0000 7f605959a640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:35:13.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.962+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f605c0b28a0 con 0x7f605c13f710 2024-03-25T19:35:13.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4220530707 0 0) 0x7f6044003520 con 0x7f605c0a98f0 2024-03-25T19:35:13.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 >> v1:172.21.15.134:6790/0 conn(0x7f605c1430c0 legacy=0x7f605c14dc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:13.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 >> v1:172.21.15.134:6789/0 conn(0x7f605c13f710 legacy=0x7f605c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:13.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 10 monclient: _finish_hunting 0 2024-03-25T19:35:13.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 1 monclient: found mon.noname-c 2024-03-25T19:35:13.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c0b44c0 con 0x7f605c0a98f0 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 10 monclient: _finish_auth 0 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 10 monclient: _check_auth_tickets 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f604c001690 con 0x7f605c0a98f0 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f6060f8f640 5 monclient: authenticate success, global_id 4510 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f6060f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f605959a640 10 monclient: dump: 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:13.978 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:13.979 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:13.979 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:13.979 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:13.979 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:13.979 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:13.979 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:13.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.977+0000 7f6060f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:13.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f604c0017f0 con 0x7f605c0a98f0 2024-03-25T19:35:13.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f605959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f605959a640 1 -- 172.21.15.134:0/4086464341 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f604c001c00 con 0x7f605c0a98f0 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f605959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f605959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f605959a640 10 monclient: dump: 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:13.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f6060f8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:13.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.991+0000 7f6060f8f640 4 set_mon_vals no callback set 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.992+0000 7f6060f8f640 10 monclient: shutdown 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.992+0000 7f6060f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.992+0000 7f6060f8f640 1 -- 172.21.15.134:0/4086464341 >> v1:172.21.15.159:6789/0 conn(0x7f605c0a98f0 legacy=0x7f605c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.992+0000 7f6060f8f640 1 -- 172.21.15.134:0/4086464341 shutdown_connections 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.992+0000 7f6060f8f640 1 -- 172.21.15.134:0/4086464341 >> 172.21.15.134:0/4086464341 conn(0x7f605c0b7ea0 msgr2=0x7f605c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.992+0000 7f6060f8f640 1 -- 172.21.15.134:0/4086464341 shutdown_connections 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.992+0000 7f6060f8f640 1 -- 172.21.15.134:0/4086464341 wait complete. 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.993+0000 7f6060f8f640 10 monclient: build_initial_monmap 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.993+0000 7f6060f8f640 10 monclient: monmap: 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:13.994 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.993+0000 7f6060f8f640 1 librados: starting msgr at 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.993+0000 7f6060f8f640 1 librados: starting objecter 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.993+0000 7f6060f8f640 1 Processor -- start 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.993+0000 7f6060f8f640 1 -- start start 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.994+0000 7f6060f8f640 1 librados: setting wanted keys 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.994+0000 7f6060f8f640 1 librados: calling monclient init 2024-03-25T19:35:13.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.994+0000 7f6060f8f640 10 monclient: init 2024-03-25T19:35:13.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.994+0000 7f6060f8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:13.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.994+0000 7f6060f8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.994+0000 7f6060f8f640 10 monclient(hunting): picked mon.noname-b con 0x7f605c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.994+0000 7f6060f8f640 10 monclient(hunting): picked mon.noname-a con 0x7f605c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.995+0000 7f6060f8f640 10 monclient(hunting): picked mon.noname-c con 0x7f605c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.995+0000 7f6060f8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c1d41e0 con 0x7f605c13f710 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.995+0000 7f6060f8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c0ab950 con 0x7f605c0a98f0 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.995+0000 7f6060f8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c14e5c0 con 0x7f605c1430c0 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.995+0000 7f6060f8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:13.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:13.995+0000 7f6060f8f640 10 monclient(hunting): authenticate will time out at 6346.153745s 2024-03-25T19:35:14.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.001+0000 7f605a59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f605c0a98f0 0x7f605c138250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:41768/0 (socket says 172.21.15.134:41768) 2024-03-25T19:35:14.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.001+0000 7f605a59c640 1 -- 172.21.15.134:0/458777330 learned_addr learned my addr 172.21.15.134:0/458777330 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:14.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.008+0000 7f604affd640 10 client.?.objecter ms_handle_connect 0x7f605c0a98f0 2024-03-25T19:35:14.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.008+0000 7f604affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:14.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.009+0000 7f604affd640 10 client.?.objecter ms_handle_connect 0x7f605c13f710 2024-03-25T19:35:14.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.009+0000 7f604affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:14.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427197257 0 0) 0x7f605c0ab950 con 0x7f605c0a98f0 2024-03-25T19:35:14.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:14.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:14.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:14.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:35:14.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c004eb0 con 0x7f605c0a98f0 2024-03-25T19:35:14.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 10 client.?.objecter ms_handle_connect 0x7f605c1430c0 2024-03-25T19:35:14.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.020+0000 7f604affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:14.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.027+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239435724 0 0) 0x7f605c1d41e0 con 0x7f605c13f710 2024-03-25T19:35:14.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.027+0000 7f604affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:14.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.027+0000 7f604affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:14.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.027+0000 7f604affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:14.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.027+0000 7f604affd640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:35:14.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.027+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f605c0ab950 con 0x7f605c13f710 2024-03-25T19:35:14.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.032+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3262967714 0 0) 0x7f603c004eb0 con 0x7f605c0a98f0 2024-03-25T19:35:14.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.032+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f605c1d41e0 con 0x7f605c0a98f0 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.032+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f604c001690 con 0x7f605c0a98f0 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.032+0000 7f604affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.032+0000 7f604affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.032+0000 7f604affd640 10 monclient(hunting): dump: 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:14.033 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:14.034 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:14.034 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:14.034 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:14.034 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:14.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.032+0000 7f604affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:14.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.034+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028226533 0 0) 0x7f605c14e5c0 con 0x7f605c1430c0 2024-03-25T19:35:14.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.034+0000 7f604affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:14.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.034+0000 7f604affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:14.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.034+0000 7f604affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:14.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.034+0000 7f604affd640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:35:14.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.034+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c004eb0 con 0x7f605c1430c0 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2159793187 0 0) 0x7f605c1d41e0 con 0x7f605c0a98f0 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 >> v1:172.21.15.134:6790/0 conn(0x7f605c1430c0 legacy=0x7f605c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 >> v1:172.21.15.134:6789/0 conn(0x7f605c13f710 legacy=0x7f605c1d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 1 monclient: found mon.b 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:14.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c14e750 con 0x7f605c0a98f0 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 10 monclient: _finish_auth 0 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f604affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 10 monclient: _renew_subs 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c138960 con 0x7f605c0a98f0 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 10 monclient: _renew_subs 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:14.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c1d82d0 con 0x7f605c0a98f0 2024-03-25T19:35:14.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.039+0000 7f6060f8f640 1 librados: init done 2024-03-25T19:35:14.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.040+0000 7f6060f8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:14.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.040+0000 7f6060f8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:14.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.040+0000 7f6060f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:14.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.040+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c0627b0 con 0x7f605c0a98f0 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.311+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f604c0038a0 con 0x7f605c0a98f0 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.312+0000 7f604affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.312+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f604c001cc0 con 0x7f605c0a98f0 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.312+0000 7f604affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.312+0000 7f604affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.312+0000 7f604affd640 10 monclient: dump: 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:14.312 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:14.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.312+0000 7f6058d99640 4 set_mon_vals no callback set 2024-03-25T19:35:14.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.313+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f604c003db0 con 0x7f605c0a98f0 2024-03-25T19:35:14.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.313+0000 7f6059d9b640 10 monclient: get_auth_request con 0x7f603c06de00 auth_method 0 2024-03-25T19:35:14.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.313+0000 7f604affd640 10 client.4513.objecter ms_handle_connect 0x7f603c06de00 2024-03-25T19:35:14.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f604c054910 con 0x7f605c0a98f0 2024-03-25T19:35:14.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 10 client.4513.objecter ms_dispatch 0x7f605c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:14.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:14.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:14.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T19:35:14.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f604c050420 con 0x7f605c0a98f0 2024-03-25T19:35:14.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:14.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.318+0000 7f604affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:14.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.546+0000 7f6060f8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:14.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.546+0000 7f6060f8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:14.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.546+0000 7f6060f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:14.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.546+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f605c13dc00 con 0x7f605c0a98f0 2024-03-25T19:35:14.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.658+0000 7f604affd640 1 -- 172.21.15.134:0/458777330 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f604c052350 con 0x7f605c0a98f0 2024-03-25T19:35:14.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.658+0000 7f604affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:14.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.658+0000 7f604affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:14.659 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:35:14.659 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:14.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.661+0000 7f6060f8f640 10 librados: watch_flush enter 2024-03-25T19:35:14.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.661+0000 7f6060f8f640 10 librados: watch_flush exit 2024-03-25T19:35:14.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.661+0000 7f6060f8f640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T19:35:14.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.661+0000 7f6060f8f640 10 client.4513.objecter successfully canceled tick 2024-03-25T19:35:14.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 >> v1:172.21.15.159:6800/1875874887 conn(0x7f603c06de00 legacy=0x7f603c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:14.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 10 monclient: shutdown 2024-03-25T19:35:14.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:14.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 >> v1:172.21.15.159:6789/0 conn(0x7f605c0a98f0 legacy=0x7f605c138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:14.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 shutdown_connections 2024-03-25T19:35:14.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 >> 172.21.15.134:0/458777330 conn(0x7f605c0b7ea0 msgr2=0x7f605c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:14.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 shutdown_connections 2024-03-25T19:35:14.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 1 -- 172.21.15.134:0/458777330 wait complete. 2024-03-25T19:35:14.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:14.662+0000 7f6060f8f640 1 librados: shutdown 2024-03-25T19:35:14.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:21.481 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:21.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.600+0000 7f53a13fb640 10 monclient: get_monmap_and_config 2024-03-25T19:35:21.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.600+0000 7f53a13fb640 10 monclient: build_initial_monmap 2024-03-25T19:35:21.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.600+0000 7f53a13fb640 10 monclient: monmap: 2024-03-25T19:35:21.600 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:21.600 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:21.600 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:21.601464+0000 2024-03-25T19:35:21.600 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:21.601464+0000 2024-03-25T19:35:21.601 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:21.601 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:21.601 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:21.601 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:21.601 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:21.601 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.601+0000 7f53a13fb640 1 Processor -- start 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.601+0000 7f53a13fb640 1 -- start start 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.601+0000 7f53a13fb640 10 monclient: init 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 10 monclient(hunting): picked mon.noname-a con 0x7f539c06f360 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 10 monclient(hunting): picked mon.noname-b con 0x7f539c06f740 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 10 monclient(hunting): picked mon.noname-c con 0x7f539c142e30 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c071d20 con 0x7f539c06f360 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c071340 con 0x7f539c142e30 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c071620 con 0x7f539c06f740 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:21.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.602+0000 7f53a13fb640 10 monclient(hunting): authenticate will time out at 6353.761384s 2024-03-25T19:35:21.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.610+0000 7f539affd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f539c06f360 0x7f539c142590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:32880/0 (socket says 172.21.15.134:32880) 2024-03-25T19:35:21.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.610+0000 7f539affd640 1 -- 172.21.15.134:0/1887927732 learned_addr learned my addr 172.21.15.134:0/1887927732 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:21.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.620+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898470472 0 0) 0x7f539c071d20 con 0x7f539c06f360 2024-03-25T19:35:21.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.620+0000 7f5399ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:21.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.620+0000 7f5399ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:21.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.620+0000 7f5399ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:21.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.620+0000 7f5399ffb640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:35:21.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.620+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5384005730 con 0x7f539c06f360 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.625+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246196680 0 0) 0x7f539c071340 con 0x7f539c142e30 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.625+0000 7f5399ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.625+0000 7f5399ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.625+0000 7f5399ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.625+0000 7f5399ffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.625+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f539c071d20 con 0x7f539c142e30 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.626+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526171644 0 0) 0x7f539c071620 con 0x7f539c06f740 2024-03-25T19:35:21.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.626+0000 7f5399ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:21.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.626+0000 7f5399ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:21.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.626+0000 7f5399ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:21.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.626+0000 7f5399ffb640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:35:21.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.626+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f539c071340 con 0x7f539c06f740 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.636+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1190528166 0 0) 0x7f539c071d20 con 0x7f539c142e30 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.636+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 >> v1:172.21.15.134:6790/0 conn(0x7f539c06f740 legacy=0x7f539c072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.636+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 >> v1:172.21.15.134:6789/0 conn(0x7f539c06f360 legacy=0x7f539c142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.636+0000 7f5399ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.636+0000 7f5399ffb640 1 monclient: found mon.noname-c 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c071900 con 0x7f539c142e30 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:21.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f53940019f0 con 0x7f539c142e30 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f5399ffb640 10 monclient: dump: 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:21.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f53a13fb640 5 monclient: authenticate success, global_id 4516 2024-03-25T19:35:21.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.637+0000 7f53a13fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:21.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5394001c10 con 0x7f539c142e30 2024-03-25T19:35:21.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f5399ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:21.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f5399ffb640 1 -- 172.21.15.134:0/1887927732 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f5394001f20 con 0x7f539c142e30 2024-03-25T19:35:21.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f5399ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f5399ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f5399ffb640 10 monclient: dump: 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f53a13fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:21.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.650+0000 7f53a13fb640 4 set_mon_vals no callback set 2024-03-25T19:35:21.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.651+0000 7f53a13fb640 10 monclient: shutdown 2024-03-25T19:35:21.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.651+0000 7f53a13fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:21.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.651+0000 7f53a13fb640 1 -- 172.21.15.134:0/1887927732 >> v1:172.21.15.159:6789/0 conn(0x7f539c142e30 legacy=0x7f539c14a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.651+0000 7f53a13fb640 1 -- 172.21.15.134:0/1887927732 shutdown_connections 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.651+0000 7f53a13fb640 1 -- 172.21.15.134:0/1887927732 >> 172.21.15.134:0/1887927732 conn(0x7f539c0b4b80 msgr2=0x7f539c0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.651+0000 7f53a13fb640 1 -- 172.21.15.134:0/1887927732 shutdown_connections 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.651+0000 7f53a13fb640 1 -- 172.21.15.134:0/1887927732 wait complete. 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.652+0000 7f53a13fb640 10 monclient: build_initial_monmap 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.652+0000 7f53a13fb640 10 monclient: monmap: 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.653+0000 7f53a13fb640 1 librados: starting msgr at 2024-03-25T19:35:21.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.653+0000 7f53a13fb640 1 librados: starting objecter 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.653+0000 7f53a13fb640 1 Processor -- start 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.653+0000 7f53a13fb640 1 -- start start 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.653+0000 7f53a13fb640 1 librados: setting wanted keys 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.653+0000 7f53a13fb640 1 librados: calling monclient init 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.653+0000 7f53a13fb640 10 monclient: init 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 10 monclient(hunting): picked mon.noname-c con 0x7f539c06f360 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 10 monclient(hunting): picked mon.noname-a con 0x7f539c06f740 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:21.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 10 monclient(hunting): picked mon.noname-b con 0x7f539c142e30 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:21.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c1d0b30 con 0x7f539c06f740 2024-03-25T19:35:21.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c13d540 con 0x7f539c142e30 2024-03-25T19:35:21.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c13f930 con 0x7f539c06f360 2024-03-25T19:35:21.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.654+0000 7f53a13fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:21.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.655+0000 7f53a13fb640 10 monclient(hunting): authenticate will time out at 6353.813639s 2024-03-25T19:35:21.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.657+0000 7f539affd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f539c06f360 0x7f539c0ad6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:41302/0 (socket says 172.21.15.134:41302) 2024-03-25T19:35:21.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.657+0000 7f539affd640 1 -- 172.21.15.134:0/4047402194 learned_addr learned my addr 172.21.15.134:0/4047402194 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:21.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.658+0000 7f53837fe640 10 client.?.objecter ms_handle_connect 0x7f539c06f360 2024-03-25T19:35:21.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.658+0000 7f53837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:21.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.662+0000 7f53837fe640 10 client.?.objecter ms_handle_connect 0x7f539c06f740 2024-03-25T19:35:21.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.662+0000 7f53837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:21.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.672+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262605916 0 0) 0x7f539c13f930 con 0x7f539c06f360 2024-03-25T19:35:21.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.672+0000 7f53837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:21.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.672+0000 7f53837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:21.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.672+0000 7f53837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:21.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.672+0000 7f53837fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:35:21.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.672+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5388004eb0 con 0x7f539c06f360 2024-03-25T19:35:21.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.677+0000 7f53837fe640 10 client.?.objecter ms_handle_connect 0x7f539c142e30 2024-03-25T19:35:21.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.677+0000 7f53837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.678+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643029690 0 0) 0x7f5388004eb0 con 0x7f539c06f360 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.678+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f539c13f930 con 0x7f539c06f360 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.678+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f538c001950 con 0x7f539c06f360 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): dump: 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.679 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201594950 0 0) 0x7f539c071340 con 0x7f539c06f740 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:35:21.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.679+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5388004eb0 con 0x7f539c06f740 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3029081610 0 0) 0x7f539c13f930 con 0x7f539c06f360 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 >> v1:172.21.15.159:6789/0 conn(0x7f539c142e30 legacy=0x7f539c1d2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 >> v1:172.21.15.134:6789/0 conn(0x7f539c06f740 legacy=0x7f539c0addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 1 monclient: found mon.c 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c13ce30 con 0x7f539c06f360 2024-03-25T19:35:21.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 5 monclient: authenticate success, global_id 4523 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 10 monclient: _renew_subs 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539c13da00 con 0x7f539c06f360 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 10 monclient: _renew_subs 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f539c13e310 con 0x7f539c06f360 2024-03-25T19:35:21.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.688+0000 7f53a13fb640 1 librados: init done 2024-03-25T19:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.689+0000 7f53a13fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.689+0000 7f53a13fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.689+0000 7f53a13fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.689+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539c13dc40 con 0x7f539c06f360 2024-03-25T19:35:21.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.993+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f538c0047e0 con 0x7f539c06f360 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.993+0000 7f53837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.993+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f538c0049d0 con 0x7f539c06f360 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.993+0000 7f53837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.993+0000 7f53837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.993+0000 7f53837fe640 10 monclient: dump: 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:21.994 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:21.995 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:21.995 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:21.995 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:21.995 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:21.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.993+0000 7f53997fa640 4 set_mon_vals no callback set 2024-03-25T19:35:21.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.994+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f538c004bf0 con 0x7f539c06f360 2024-03-25T19:35:21.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.995+0000 7f539a7fc640 10 monclient: get_auth_request con 0x7f538806dd90 auth_method 0 2024-03-25T19:35:21.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:21.995+0000 7f53837fe640 10 client.4523.objecter ms_handle_connect 0x7f538806dd90 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.000+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f538c054670 con 0x7f539c06f360 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.000+0000 7f53837fe640 10 client.4523.objecter ms_dispatch 0x7f539c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.000+0000 7f53837fe640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.000+0000 7f53837fe640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.001+0000 7f53837fe640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.001+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f538c0b18b0 con 0x7f539c06f360 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.001+0000 7f53837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:22.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.001+0000 7f53837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:22.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.251+0000 7f53a13fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:22.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.251+0000 7f53a13fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:22.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.251+0000 7f53a13fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:22.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.251+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f539c0627b0 con 0x7f539c06f360 2024-03-25T19:35:22.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.275+0000 7f53837fe640 1 -- 172.21.15.134:0/4047402194 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f538c050180 con 0x7f539c06f360 2024-03-25T19:35:22.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.275+0000 7f53837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:22.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.275+0000 7f53837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:22.275 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:35:22.276 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 10 librados: watch_flush enter 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 10 librados: watch_flush exit 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 10 client.4523.objecter successfully canceled tick 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 >> v1:172.21.15.159:6800/1875874887 conn(0x7f538806dd90 legacy=0x7f5388070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 10 monclient: shutdown 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 >> v1:172.21.15.134:6790/0 conn(0x7f539c06f360 legacy=0x7f539c0ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 shutdown_connections 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 >> 172.21.15.134:0/4047402194 conn(0x7f539c0b4b80 msgr2=0x7f539c0b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:22.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 shutdown_connections 2024-03-25T19:35:22.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 1 -- 172.21.15.134:0/4047402194 wait complete. 2024-03-25T19:35:22.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:22.277+0000 7f53a13fb640 1 librados: shutdown 2024-03-25T19:35:22.289 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:25.811 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:30.483 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.821 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:30.493 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.849 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:30.521 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.873 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:30.545 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.876 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:30.548 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.883 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:30.530 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.892 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:30.539 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.917 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:30.564 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:25.924 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:30.596 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:29.095 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:29.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.209+0000 7fd0efc25640 10 monclient: get_monmap_and_config 2024-03-25T19:35:29.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.209+0000 7fd0efc25640 10 monclient: build_initial_monmap 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.209+0000 7fd0efc25640 10 monclient: monmap: 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:29.210744+0000 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:29.210744+0000 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:29.210 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:29.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.210+0000 7fd0efc25640 1 Processor -- start 2024-03-25T19:35:29.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.210+0000 7fd0efc25640 1 -- start start 2024-03-25T19:35:29.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.210+0000 7fd0efc25640 10 monclient: init 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e8142520 con 0x7fd0e80a98f0 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e80b28a0 con 0x7fd0e81430c0 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e80b2a30 con 0x7fd0e813f710 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:29.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.211+0000 7fd0efc25640 10 monclient(hunting): authenticate will time out at 6361.370384s 2024-03-25T19:35:29.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.222+0000 7fd0ed9c1640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd0e80a98f0 0x7fd0e80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:37640/0 (socket says 172.21.15.134:37640) 2024-03-25T19:35:29.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.222+0000 7fd0ed9c1640 1 -- 172.21.15.134:0/1268247912 learned_addr learned my addr 172.21.15.134:0/1268247912 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059775298 0 0) 0x7fd0e8142520 con 0x7fd0e80a98f0 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d4003520 con 0x7fd0e80a98f0 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904861011 0 0) 0x7fd0e80b2a30 con 0x7fd0e813f710 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:35:29.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.242+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e8142520 con 0x7fd0e813f710 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469240923 0 0) 0x7fd0d4003520 con 0x7fd0e80a98f0 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 >> v1:172.21.15.134:6790/0 conn(0x7fd0e813f710 legacy=0x7fd0e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 >> v1:172.21.15.159:6789/0 conn(0x7fd0e81430c0 legacy=0x7fd0e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 10 monclient: _finish_hunting 0 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 1 monclient: found mon.noname-a 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e80b44c0 con 0x7fd0e80a98f0 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 10 monclient: _finish_auth 0 2024-03-25T19:35:29.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 10 monclient: _check_auth_tickets 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd0e4001690 con 0x7fd0e80a98f0 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0ec9bf640 10 monclient: dump: 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:29.257 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:29.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0efc25640 5 monclient: authenticate success, global_id 4563 2024-03-25T19:35:29.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.256+0000 7fd0efc25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:29.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0e40017f0 con 0x7fd0e80a98f0 2024-03-25T19:35:29.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0ec9bf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:29.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0ec9bf640 1 -- 172.21.15.134:0/1268247912 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd0e4001c00 con 0x7fd0e80a98f0 2024-03-25T19:35:29.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0ec9bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0ec9bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0ec9bf640 10 monclient: dump: 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0efc25640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:29.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.267+0000 7fd0efc25640 4 set_mon_vals no callback set 2024-03-25T19:35:29.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.268+0000 7fd0efc25640 10 monclient: shutdown 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.268+0000 7fd0efc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.268+0000 7fd0efc25640 1 -- 172.21.15.134:0/1268247912 >> v1:172.21.15.134:6789/0 conn(0x7fd0e80a98f0 legacy=0x7fd0e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.268+0000 7fd0efc25640 1 -- 172.21.15.134:0/1268247912 shutdown_connections 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.268+0000 7fd0efc25640 1 -- 172.21.15.134:0/1268247912 >> 172.21.15.134:0/1268247912 conn(0x7fd0e80b7ea0 msgr2=0x7fd0e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.268+0000 7fd0efc25640 1 -- 172.21.15.134:0/1268247912 shutdown_connections 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.268+0000 7fd0efc25640 1 -- 172.21.15.134:0/1268247912 wait complete. 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.269+0000 7fd0efc25640 10 monclient: build_initial_monmap 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.269+0000 7fd0efc25640 10 monclient: monmap: 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:29.270 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 1 librados: starting msgr at 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 1 librados: starting objecter 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 1 Processor -- start 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 1 -- start start 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 1 librados: setting wanted keys 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 1 librados: calling monclient init 2024-03-25T19:35:29.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 10 monclient: init 2024-03-25T19:35:29.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:29.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:29.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e80a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:29.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.270+0000 7fd0efc25640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e813f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:29.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.271+0000 7fd0efc25640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:29.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.271+0000 7fd0efc25640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e80acc10 con 0x7fd0e813f710 2024-03-25T19:35:29.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.271+0000 7fd0efc25640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e80ace80 con 0x7fd0e80a98f0 2024-03-25T19:35:29.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.271+0000 7fd0efc25640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e81399d0 con 0x7fd0e81430c0 2024-03-25T19:35:29.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.271+0000 7fd0efc25640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:29.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.271+0000 7fd0efc25640 10 monclient(hunting): authenticate will time out at 6361.429873s 2024-03-25T19:35:29.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.277+0000 7fd0ed9c1640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd0e80a98f0 0x7fd0e80ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:47930/0 (socket says 172.21.15.134:47930) 2024-03-25T19:35:29.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.277+0000 7fd0ed9c1640 1 -- 172.21.15.134:0/1318307844 learned_addr learned my addr 172.21.15.134:0/1318307844 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:29.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.278+0000 7fd0de7fc640 10 client.?.objecter ms_handle_connect 0x7fd0e80a98f0 2024-03-25T19:35:29.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.278+0000 7fd0de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.284+0000 7fd0de7fc640 10 client.?.objecter ms_handle_connect 0x7fd0e81430c0 2024-03-25T19:35:29.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.284+0000 7fd0de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.288+0000 7fd0de7fc640 10 client.?.objecter ms_handle_connect 0x7fd0e813f710 2024-03-25T19:35:29.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.288+0000 7fd0de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.346+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538274432 0 0) 0x7fd0e80acc10 con 0x7fd0e813f710 2024-03-25T19:35:29.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.346+0000 7fd0de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.346+0000 7fd0de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.346+0000 7fd0de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.346+0000 7fd0de7fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:35:29.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.346+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0cc004eb0 con 0x7fd0e813f710 2024-03-25T19:35:29.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.367+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2472428892 0 0) 0x7fd0cc004eb0 con 0x7fd0e813f710 2024-03-25T19:35:29.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.367+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0e80acc10 con 0x7fd0e813f710 2024-03-25T19:35:29.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.367+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd0d8001930 con 0x7fd0e813f710 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.367+0000 7fd0de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.367+0000 7fd0de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.367+0000 7fd0de7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:29.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.367+0000 7fd0de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1676077091 0 0) 0x7fd0e80acc10 con 0x7fd0e813f710 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 >> v1:172.21.15.134:6790/0 conn(0x7fd0e81430c0 legacy=0x7fd0e81e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 >> v1:172.21.15.159:6789/0 conn(0x7fd0e80a98f0 legacy=0x7fd0e80ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 1 monclient: found mon.a 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e80ad9b0 con 0x7fd0e813f710 2024-03-25T19:35:29.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 5 monclient: authenticate success, global_id 4566 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 10 monclient: _renew_subs 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e8071250 con 0x7fd0e813f710 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 10 monclient: _renew_subs 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e80abb50 con 0x7fd0e813f710 2024-03-25T19:35:29.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.380+0000 7fd0efc25640 1 librados: init done 2024-03-25T19:35:29.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.381+0000 7fd0efc25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.381+0000 7fd0efc25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.381+0000 7fd0efc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:35:29.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.381+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e80abe50 con 0x7fd0e813f710 2024-03-25T19:35:29.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.669+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0d8003610 con 0x7fd0e813f710 2024-03-25T19:35:29.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.669+0000 7fd0de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:29.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.669+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd0d8003a60 con 0x7fd0e813f710 2024-03-25T19:35:29.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.669+0000 7fd0de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.669+0000 7fd0de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:29.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.669+0000 7fd0de7fc640 10 monclient: dump: 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:29.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.669+0000 7fd0dffff640 4 set_mon_vals no callback set 2024-03-25T19:35:29.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.670+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fd0d8003c80 con 0x7fd0e813f710 2024-03-25T19:35:29.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.671+0000 7fd0ed9c1640 10 monclient: get_auth_request con 0x7fd0cc06dc70 auth_method 0 2024-03-25T19:35:29.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.671+0000 7fd0de7fc640 10 client.4566.objecter ms_handle_connect 0x7fd0cc06dc70 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fd0d80535f0 con 0x7fd0e813f710 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 10 client.4566.objecter ms_dispatch 0x7fd0e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0d804f100 con 0x7fd0e813f710 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.679+0000 7fd0de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:29.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.933+0000 7fd0efc25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:29.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.933+0000 7fd0efc25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:29.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.933+0000 7fd0efc25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:35:29.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.933+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0e80627b0 con 0x7fd0e813f710 2024-03-25T19:35:29.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.961+0000 7fd0de7fc640 1 -- 172.21.15.134:0/1318307844 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fd0d8051030 con 0x7fd0e813f710 2024-03-25T19:35:29.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.961+0000 7fd0de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:29.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.961+0000 7fd0de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:29.962 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:35:29.962 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:29.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 10 librados: watch_flush enter 2024-03-25T19:35:29.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 10 librados: watch_flush exit 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 10 client.4566.objecter successfully canceled tick 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 >> v1:172.21.15.159:6800/1875874887 conn(0x7fd0cc06dc70 legacy=0x7fd0cc070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 10 monclient: shutdown 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 >> v1:172.21.15.134:6789/0 conn(0x7fd0e813f710 legacy=0x7fd0e806f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 shutdown_connections 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 >> 172.21.15.134:0/1318307844 conn(0x7fd0e80b7ea0 msgr2=0x7fd0e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 shutdown_connections 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 1 -- 172.21.15.134:0/1318307844 wait complete. 2024-03-25T19:35:29.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:29.963+0000 7fd0efc25640 1 librados: shutdown 2024-03-25T19:35:29.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:36.784 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:36.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.901+0000 7fdbcb688640 10 monclient: get_monmap_and_config 2024-03-25T19:35:36.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.901+0000 7fdbcb688640 10 monclient: build_initial_monmap 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.901+0000 7fdbcb688640 10 monclient: monmap: 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:36.902759+0000 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:36.902759+0000 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:36.902 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:36.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.902+0000 7fdbcb688640 1 Processor -- start 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.902+0000 7fdbcb688640 1 -- start start 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.902+0000 7fdbcb688640 10 monclient: init 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc41430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc4142520 con 0x7fdbc40a98f0 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40b28a0 con 0x7fdbc41430c0 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40b2a30 con 0x7fdbc413f710 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:36.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.903+0000 7fdbcb688640 10 monclient(hunting): authenticate will time out at 6369.062476s 2024-03-25T19:35:36.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.916+0000 7fdbc9c25640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fdbc41430c0 0x7fdbc414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:53482/0 (socket says 172.21.15.134:53482) 2024-03-25T19:35:36.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.916+0000 7fdbc9c25640 1 -- 172.21.15.134:0/868787258 learned_addr learned my addr 172.21.15.134:0/868787258 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:36.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.925+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219619370 0 0) 0x7fdbc40b28a0 con 0x7fdbc41430c0 2024-03-25T19:35:36.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.925+0000 7fdbbbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:36.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.925+0000 7fdbbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:36.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.925+0000 7fdbbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:36.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.925+0000 7fdbbbfff640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:35:36.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.925+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb0003520 con 0x7fdbc41430c0 2024-03-25T19:35:36.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.932+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836744234 0 0) 0x7fdbc4142520 con 0x7fdbc40a98f0 2024-03-25T19:35:36.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.932+0000 7fdbbbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:36.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.932+0000 7fdbbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:36.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.932+0000 7fdbbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:36.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.932+0000 7fdbbbfff640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:35:36.932 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.932+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc40b28a0 con 0x7fdbc40a98f0 2024-03-25T19:35:36.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934671615 0 0) 0x7fdbb0003520 con 0x7fdbc41430c0 2024-03-25T19:35:36.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 >> v1:172.21.15.134:6790/0 conn(0x7fdbc413f710 legacy=0x7fdbc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:36.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 >> v1:172.21.15.134:6789/0 conn(0x7fdbc40a98f0 legacy=0x7fdbc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:36.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:36.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 1 monclient: found mon.noname-c 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40b44c0 con 0x7fdbc41430c0 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 10 monclient: _finish_auth 0 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdbbc001a10 con 0x7fdbc41430c0 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbbbfff640 10 monclient: dump: 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:36.935 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbcb688640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:35:36.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.934+0000 7fdbcb688640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:36.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbbc001b70 con 0x7fdbc41430c0 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbbbfff640 1 -- 172.21.15.134:0/868787258 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdbbc001f80 con 0x7fdbc41430c0 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbbbfff640 10 monclient: dump: 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:36.941 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:36.942 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:36.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbcb688640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:36.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbcb688640 4 set_mon_vals no callback set 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbcb688640 10 monclient: shutdown 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbcb688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbcb688640 1 -- 172.21.15.134:0/868787258 >> v1:172.21.15.159:6789/0 conn(0x7fdbc41430c0 legacy=0x7fdbc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbcb688640 1 -- 172.21.15.134:0/868787258 shutdown_connections 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.941+0000 7fdbcb688640 1 -- 172.21.15.134:0/868787258 >> 172.21.15.134:0/868787258 conn(0x7fdbc40b7ea0 msgr2=0x7fdbc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.942+0000 7fdbcb688640 1 -- 172.21.15.134:0/868787258 shutdown_connections 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.942+0000 7fdbcb688640 1 -- 172.21.15.134:0/868787258 wait complete. 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient: build_initial_monmap 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient: monmap: 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 1 librados: starting msgr at 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 1 librados: starting objecter 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 1 Processor -- start 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 1 -- start start 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 1 librados: setting wanted keys 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 1 librados: calling monclient init 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient: init 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:36.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.943+0000 7fdbcb688640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc41430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.944+0000 7fdbcb688640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc41d7bc0 con 0x7fdbc40a98f0 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.944+0000 7fdbcb688640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc41d7750 con 0x7fdbc41430c0 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.944+0000 7fdbcb688640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc41d78e0 con 0x7fdbc413f710 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.944+0000 7fdbcb688640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:36.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.944+0000 7fdbcb688640 10 monclient(hunting): authenticate will time out at 6369.102662s 2024-03-25T19:35:36.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.962+0000 7fdbc9c25640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fdbc41430c0 0x7fdbc41d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:53538/0 (socket says 172.21.15.134:53538) 2024-03-25T19:35:36.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.962+0000 7fdbc9424640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fdbc40a98f0 0x7fdbc41d8050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:33394/0 (socket says 172.21.15.134:33394) 2024-03-25T19:35:36.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.962+0000 7fdbc9c25640 1 -- 172.21.15.134:0/380448793 learned_addr learned my addr 172.21.15.134:0/380448793 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.963+0000 7fdbb9ffb640 10 client.?.objecter ms_handle_connect 0x7fdbc41430c0 2024-03-25T19:35:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.963+0000 7fdbb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.963+0000 7fdbb9ffb640 10 client.?.objecter ms_handle_connect 0x7fdbc40a98f0 2024-03-25T19:35:36.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.963+0000 7fdbb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:36.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.967+0000 7fdbb9ffb640 10 client.?.objecter ms_handle_connect 0x7fdbc413f710 2024-03-25T19:35:36.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.967+0000 7fdbb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.973+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554525886 0 0) 0x7fdbc41d7bc0 con 0x7fdbc40a98f0 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.973+0000 7fdbb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.973+0000 7fdbb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.973+0000 7fdbb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.973+0000 7fdbb9ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.974+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba8004eb0 con 0x7fdbc40a98f0 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.974+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101143569 0 0) 0x7fdbc41d7750 con 0x7fdbc41430c0 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.974+0000 7fdbb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:36.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.974+0000 7fdbb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:36.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.974+0000 7fdbb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:36.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.974+0000 7fdbb9ffb640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:35:36.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.974+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc41d7bc0 con 0x7fdbc41430c0 2024-03-25T19:35:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428719458 0 0) 0x7fdbc41d7bc0 con 0x7fdbc41430c0 2024-03-25T19:35:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc41d7750 con 0x7fdbc41430c0 2024-03-25T19:35:36.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdbbc001f40 con 0x7fdbc41430c0 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:36.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971495838 0 0) 0x7fdbc41d78e0 con 0x7fdbc413f710 2024-03-25T19:35:36.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:36.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:36.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:36.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:35:36.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.980+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc41d7bc0 con 0x7fdbc413f710 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3425496437 0 0) 0x7fdbc41d7bc0 con 0x7fdbc413f710 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc41d78e0 con 0x7fdbc413f710 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdbb4001860 con 0x7fdbc413f710 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:36.995 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2081114952 0 0) 0x7fdbc41d7750 con 0x7fdbc41430c0 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 >> v1:172.21.15.134:6790/0 conn(0x7fdbc413f710 legacy=0x7fdbc41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 >> v1:172.21.15.134:6789/0 conn(0x7fdbc40a98f0 legacy=0x7fdbc41d8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 1 monclient: found mon.b 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.994+0000 7fdbb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbb9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc4135f80 con 0x7fdbc41430c0 2024-03-25T19:35:36.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbb9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 5 monclient: authenticate success, global_id 4528 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 10 monclient: _renew_subs 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc413c990 con 0x7fdbc41430c0 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 10 monclient: _renew_subs 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc41d51e0 con 0x7fdbc41430c0 2024-03-25T19:35:36.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.995+0000 7fdbcb688640 1 librados: init done 2024-03-25T19:35:36.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.996+0000 7fdbcb688640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:36.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.996+0000 7fdbcb688640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:36.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.996+0000 7fdbcb688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:36.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:36.996+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc4136170 con 0x7fdbc41430c0 2024-03-25T19:35:37.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.206+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbbc004e10 con 0x7fdbc41430c0 2024-03-25T19:35:37.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.206+0000 7fdbb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:37.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.206+0000 7fdbbb7fe640 4 set_mon_vals no callback set 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.278+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdbbc002100 con 0x7fdbc41430c0 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.278+0000 7fdbb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.278+0000 7fdbb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.278+0000 7fdbb9ffb640 10 monclient: dump: 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:37.278 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:37.279 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:37.279 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:37.279 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:37.279 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:37.279 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:37.279 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:37.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.279+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fdbbc01af90 con 0x7fdbc41430c0 2024-03-25T19:35:37.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.279+0000 7fdbc9424640 10 monclient: get_auth_request con 0x7fdba806df80 auth_method 0 2024-03-25T19:35:37.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.280+0000 7fdbb9ffb640 10 client.4528.objecter ms_handle_connect 0x7fdba806df80 2024-03-25T19:35:37.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.282+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fdbbc054c20 con 0x7fdbc41430c0 2024-03-25T19:35:37.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.282+0000 7fdbb9ffb640 10 client.4528.objecter ms_dispatch 0x7fdbc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:37.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.282+0000 7fdbb9ffb640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:37.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.282+0000 7fdbb9ffb640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:37.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.282+0000 7fdbb9ffb640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T19:35:37.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.285+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbbc050560 con 0x7fdbc41430c0 2024-03-25T19:35:37.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.285+0000 7fdbb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:37.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.285+0000 7fdbb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:37.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.523+0000 7fdbcb688640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:37.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.523+0000 7fdbcb688640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:37.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.523+0000 7fdbcb688640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:35:37.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.523+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbc41d7d70 con 0x7fdbc41430c0 2024-03-25T19:35:37.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.548+0000 7fdbb9ffb640 1 -- 172.21.15.134:0/380448793 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fdbc41d7d70 con 0x7fdbc41430c0 2024-03-25T19:35:37.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.549+0000 7fdbb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:37.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.549+0000 7fdbb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:37.549 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:35:37.549 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:37.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 10 librados: watch_flush enter 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 10 librados: watch_flush exit 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 10 client.4528.objecter successfully canceled tick 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 >> v1:172.21.15.159:6800/1875874887 conn(0x7fdba806df80 legacy=0x7fdba8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 10 monclient: shutdown 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 >> v1:172.21.15.159:6789/0 conn(0x7fdbc41430c0 legacy=0x7fdbc41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 shutdown_connections 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 >> 172.21.15.134:0/380448793 conn(0x7fdbc40b7ea0 msgr2=0x7fdbc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 shutdown_connections 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 1 -- 172.21.15.134:0/380448793 wait complete. 2024-03-25T19:35:37.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:37.551+0000 7fdbcb688640 1 librados: shutdown 2024-03-25T19:35:37.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:44.370 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.466+0000 7f98330f4640 10 monclient: get_monmap_and_config 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.466+0000 7f98330f4640 10 monclient: build_initial_monmap 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.466+0000 7f98330f4640 10 monclient: monmap: 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:44.467353+0000 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:44.467353+0000 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:44.466 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:44.467 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:44.467 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:44.467 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:44.467 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 1 Processor -- start 2024-03-25T19:35:44.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 1 -- start start 2024-03-25T19:35:44.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient: init 2024-03-25T19:35:44.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:44.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:44.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient(hunting): picked mon.noname-b con 0x7f982c0b10d0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:44.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient(hunting): picked mon.noname-a con 0x7f982c0b4360 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:44.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient(hunting): picked mon.noname-c con 0x7f982c0a9470 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:44.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c0a9ab0 con 0x7f982c0b4360 2024-03-25T19:35:44.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c0b2400 con 0x7f982c0a9470 2024-03-25T19:35:44.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c0aa2a0 con 0x7f982c0b10d0 2024-03-25T19:35:44.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:44.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.467+0000 7f98330f4640 10 monclient(hunting): authenticate will time out at 6376.626373s 2024-03-25T19:35:44.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.474+0000 7f9831691640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f982c0a9470 0x7f982c14dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54130/0 (socket says 172.21.15.134:54130) 2024-03-25T19:35:44.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.474+0000 7f9831691640 1 -- 172.21.15.134:0/1149959147 learned_addr learned my addr 172.21.15.134:0/1149959147 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:44.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.479+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389309596 0 0) 0x7f982c0b2400 con 0x7f982c0a9470 2024-03-25T19:35:44.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.479+0000 7f98237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:44.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.479+0000 7f98237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:44.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.479+0000 7f98237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:44.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.479+0000 7f98237fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:35:44.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.480+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9814003520 con 0x7f982c0a9470 2024-03-25T19:35:44.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354109152 0 0) 0x7f9814003520 con 0x7f982c0a9470 2024-03-25T19:35:44.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 >> v1:172.21.15.134:6790/0 conn(0x7f982c0b10d0 legacy=0x7f982c0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 >> v1:172.21.15.134:6789/0 conn(0x7f982c0b4360 legacy=0x7f982c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:44.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 1 monclient: found mon.noname-c 2024-03-25T19:35:44.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c0b14b0 con 0x7f982c0a9470 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9824001a10 con 0x7f982c0a9470 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98237fe640 10 monclient: dump: 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:44.487 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:44.488 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:44.488 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:44.488 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:44.488 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98330f4640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:35:44.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.486+0000 7f98330f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:44.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9824001b70 con 0x7f982c0a9470 2024-03-25T19:35:44.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:44.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98237fe640 1 -- 172.21.15.134:0/1149959147 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9824001f80 con 0x7f982c0a9470 2024-03-25T19:35:44.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:44.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:44.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98237fe640 10 monclient: dump: 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98330f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:44.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.498+0000 7f98330f4640 4 set_mon_vals no callback set 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.499+0000 7f98330f4640 10 monclient: shutdown 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.499+0000 7f98330f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.499+0000 7f98330f4640 1 -- 172.21.15.134:0/1149959147 >> v1:172.21.15.159:6789/0 conn(0x7f982c0a9470 legacy=0x7f982c14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.499+0000 7f98330f4640 1 -- 172.21.15.134:0/1149959147 shutdown_connections 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.499+0000 7f98330f4640 1 -- 172.21.15.134:0/1149959147 >> 172.21.15.134:0/1149959147 conn(0x7f982c0617a0 msgr2=0x7f982c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.499+0000 7f98330f4640 1 -- 172.21.15.134:0/1149959147 shutdown_connections 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.499+0000 7f98330f4640 1 -- 172.21.15.134:0/1149959147 wait complete. 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.500+0000 7f98330f4640 10 monclient: build_initial_monmap 2024-03-25T19:35:44.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.500+0000 7f98330f4640 10 monclient: monmap: 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.500+0000 7f98330f4640 1 librados: starting msgr at 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.500+0000 7f98330f4640 1 librados: starting objecter 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.500+0000 7f98330f4640 1 Processor -- start 2024-03-25T19:35:44.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 1 -- start start 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 1 librados: setting wanted keys 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 1 librados: calling monclient init 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient: init 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient(hunting): picked mon.noname-c con 0x7f982c0a9470 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient(hunting): picked mon.noname-a con 0x7f982c0b10d0 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient(hunting): picked mon.noname-b con 0x7f982c0b4360 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c1425d0 con 0x7f982c0b10d0 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c14ea80 con 0x7f982c0b4360 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c133ea0 con 0x7f982c0a9470 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:44.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.501+0000 7f98330f4640 10 monclient(hunting): authenticate will time out at 6376.660563s 2024-03-25T19:35:44.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.513+0000 7f9831691640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f982c0b4360 0x7f982c133790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54174/0 (socket says 172.21.15.134:54174) 2024-03-25T19:35:44.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.513+0000 7f9831691640 1 -- 172.21.15.134:0/1757737128 learned_addr learned my addr 172.21.15.134:0/1757737128 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:44.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.515+0000 7f98217fa640 10 client.?.objecter ms_handle_connect 0x7f982c0b4360 2024-03-25T19:35:44.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.515+0000 7f98217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:44.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.516+0000 7f98217fa640 10 client.?.objecter ms_handle_connect 0x7f982c0a9470 2024-03-25T19:35:44.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.516+0000 7f98217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:44.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.526+0000 7f98217fa640 10 client.?.objecter ms_handle_connect 0x7f982c0b10d0 2024-03-25T19:35:44.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.526+0000 7f98217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:44.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.529+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874936941 0 0) 0x7f982c133ea0 con 0x7f982c0a9470 2024-03-25T19:35:44.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.530+0000 7f98217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:44.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.530+0000 7f98217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:44.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.530+0000 7f98217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:44.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.530+0000 7f98217fa640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:35:44.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.530+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f982c0a9470 2024-03-25T19:35:44.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.535+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413277177 0 0) 0x7f982c14ea80 con 0x7f982c0b4360 2024-03-25T19:35:44.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.535+0000 7f98217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:44.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.535+0000 7f98217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:44.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.535+0000 7f98217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:44.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.535+0000 7f98217fa640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:35:44.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.535+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982c133ea0 con 0x7f982c0b4360 2024-03-25T19:35:44.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.551+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623397957 0 0) 0x7f982c1425d0 con 0x7f982c0b10d0 2024-03-25T19:35:44.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.551+0000 7f98217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:44.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.551+0000 7f98217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:44.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.551+0000 7f98217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:44.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.551+0000 7f98217fa640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:35:44.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.551+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982c14ea80 con 0x7f982c0b10d0 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.557+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089127676 0 0) 0x7f9818004eb0 con 0x7f982c0a9470 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.557+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f982c1425d0 con 0x7f982c0a9470 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.557+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9828001930 con 0x7f982c0a9470 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.557+0000 7f98217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.557+0000 7f98217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.557+0000 7f98217fa640 10 monclient(hunting): dump: 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:44.557 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:44.558 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:44.558 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.557+0000 7f98217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:44.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.562+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215237360 0 0) 0x7f982c133ea0 con 0x7f982c0b4360 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.563+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f982c0b4360 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.563+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9824001e80 con 0x7f982c0b4360 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.563+0000 7f98217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.563+0000 7f98217fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.563+0000 7f98217fa640 10 monclient(hunting): dump: 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:44.563 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:44.564 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:44.564 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:44.564 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:44.564 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.569+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3827821346 0 0) 0x7f982c14ea80 con 0x7f982c0b10d0 2024-03-25T19:35:44.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.569+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f982c133ea0 con 0x7f982c0b10d0 2024-03-25T19:35:44.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.569+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f981c001830 con 0x7f982c0b10d0 2024-03-25T19:35:44.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.569+0000 7f98217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:44.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.569+0000 7f98217fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.569+0000 7f98217fa640 10 monclient(hunting): dump: 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:44.570 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 827707070 0 0) 0x7f982c1425d0 con 0x7f982c0a9470 2024-03-25T19:35:44.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 >> v1:172.21.15.159:6789/0 conn(0x7f982c0b4360 legacy=0x7f982c133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 >> v1:172.21.15.134:6789/0 conn(0x7f982c0b10d0 legacy=0x7f982c142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 10 monclient: _finish_hunting 0 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 1 monclient: found mon.c 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c141190 con 0x7f982c0a9470 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 10 monclient: _finish_auth 0 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 10 monclient: _check_auth_tickets 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 10 monclient: _renew_subs 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c13f600 con 0x7f982c0a9470 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:44.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 10 monclient: _renew_subs 2024-03-25T19:35:44.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:44.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c1442e0 con 0x7f982c0a9470 2024-03-25T19:35:44.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.572+0000 7f98330f4640 1 librados: init done 2024-03-25T19:35:44.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.573+0000 7f98330f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:44.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.573+0000 7f98330f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:44.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.573+0000 7f98330f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:44.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.573+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c13f9a0 con 0x7f982c0a9470 2024-03-25T19:35:44.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.885+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98280035f0 con 0x7f982c0a9470 2024-03-25T19:35:44.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.885+0000 7f98217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:44.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.885+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9828003980 con 0x7f982c0a9470 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.885+0000 7f98217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.885+0000 7f98217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.885+0000 7f98217fa640 10 monclient: dump: 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:44.886 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:44.887 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:44.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.885+0000 7f9822ffd640 4 set_mon_vals no callback set 2024-03-25T19:35:44.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.886+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f9828003ba0 con 0x7f982c0a9470 2024-03-25T19:35:44.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.887+0000 7f9823fff640 10 monclient: get_auth_request con 0x7f981806e0b0 auth_method 0 2024-03-25T19:35:44.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.887+0000 7f98217fa640 10 client.4541.objecter ms_handle_connect 0x7f981806e0b0 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f9828053630 con 0x7f982c0a9470 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 10 client.4541.objecter ms_dispatch 0x7f982c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f982804f140 con 0x7f982c0a9470 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:44.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:44.894+0000 7f98217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:45.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.136+0000 7f98330f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:45.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.136+0000 7f98330f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:45.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.136+0000 7f98330f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:45.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.136+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f982c13f790 con 0x7f982c0a9470 2024-03-25T19:35:45.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.159+0000 7f98217fa640 1 -- 172.21.15.134:0/1757737128 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f9828051070 con 0x7f982c0a9470 2024-03-25T19:35:45.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.159+0000 7f98217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:45.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.159+0000 7f98217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:45.159 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:35:45.159 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.161+0000 7f98330f4640 10 librados: watch_flush enter 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.161+0000 7f98330f4640 10 librados: watch_flush exit 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.161+0000 7f98330f4640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.161+0000 7f98330f4640 10 client.4541.objecter successfully canceled tick 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 >> v1:172.21.15.159:6800/1875874887 conn(0x7f981806e0b0 legacy=0x7f9818070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 10 monclient: shutdown 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 >> v1:172.21.15.134:6790/0 conn(0x7f982c0a9470 legacy=0x7f982c131e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:45.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 shutdown_connections 2024-03-25T19:35:45.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 >> 172.21.15.134:0/1757737128 conn(0x7f982c0617a0 msgr2=0x7f982c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:45.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 shutdown_connections 2024-03-25T19:35:45.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 1 -- 172.21.15.134:0/1757737128 wait complete. 2024-03-25T19:35:45.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:45.162+0000 7f98330f4640 1 librados: shutdown 2024-03-25T19:35:45.174 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:45.907 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:50.579 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:45.912 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:50.584 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:45.948 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:50.621 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:45.968 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:50.615 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:45.976 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:50.649 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:45.978 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:50.625 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:45.984 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:50.656 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:46.010 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:50.656 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:46.018 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:50.691 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:51.980 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.095+0000 7f68cbd3d640 10 monclient: get_monmap_and_config 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.095+0000 7f68cbd3d640 10 monclient: build_initial_monmap 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.095+0000 7f68cbd3d640 10 monclient: monmap: 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:52.096326+0000 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:52.096326+0000 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:52.095 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:52.096 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:52.096 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:52.096 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.095+0000 7f68cbd3d640 1 Processor -- start 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 1 -- start start 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 10 monclient: init 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f68c40a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 10 monclient(hunting): picked mon.noname-a con 0x7f68c413f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:52.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f68c41430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:52.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.096+0000 7f68cbd3d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4142520 con 0x7f68c413f710 2024-03-25T19:35:52.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.097+0000 7f68cbd3d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c40b28a0 con 0x7f68c40a98f0 2024-03-25T19:35:52.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.097+0000 7f68cbd3d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c40b2a30 con 0x7f68c41430c0 2024-03-25T19:35:52.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.097+0000 7f68cbd3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:52.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.097+0000 7f68cbd3d640 10 monclient(hunting): authenticate will time out at 6384.255674s 2024-03-25T19:35:52.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.107+0000 7f68c9ad9640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f68c40a98f0 0x7f68c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:37220/0 (socket says 172.21.15.134:37220) 2024-03-25T19:35:52.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.107+0000 7f68c9ad9640 1 -- 172.21.15.134:0/1175005446 learned_addr learned my addr 172.21.15.134:0/1175005446 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:52.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.120+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337385291 0 0) 0x7f68c40b28a0 con 0x7f68c40a98f0 2024-03-25T19:35:52.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.120+0000 7f68c8ad7640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.120+0000 7f68c8ad7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.120+0000 7f68c8ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.120+0000 7f68c8ad7640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:35:52.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.120+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b0003520 con 0x7f68c40a98f0 2024-03-25T19:35:52.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.126+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791619293 0 0) 0x7f68c40b2a30 con 0x7f68c41430c0 2024-03-25T19:35:52.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.126+0000 7f68c8ad7640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.126+0000 7f68c8ad7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.126+0000 7f68c8ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.126+0000 7f68c8ad7640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:35:52.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.126+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c40b28a0 con 0x7f68c41430c0 2024-03-25T19:35:52.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.135+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253163737 0 0) 0x7f68c4142520 con 0x7f68c413f710 2024-03-25T19:35:52.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.135+0000 7f68c8ad7640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.135+0000 7f68c8ad7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.135+0000 7f68c8ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.135+0000 7f68c8ad7640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:35:52.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.135+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c40b2a30 con 0x7f68c413f710 2024-03-25T19:35:52.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290190794 0 0) 0x7f68b0003520 con 0x7f68c40a98f0 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 >> v1:172.21.15.134:6790/0 conn(0x7f68c41430c0 legacy=0x7f68c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 >> v1:172.21.15.134:6789/0 conn(0x7f68c413f710 legacy=0x7f68c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 10 monclient: _finish_hunting 0 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 1 monclient: found mon.noname-c 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c40b44c0 con 0x7f68c40a98f0 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 10 monclient: _finish_auth 0 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 10 monclient: _check_auth_tickets 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f68c0001690 con 0x7f68c40a98f0 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:52.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68c8ad7640 10 monclient: dump: 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:52.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68cbd3d640 5 monclient: authenticate success, global_id 4537 2024-03-25T19:35:52.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.140+0000 7f68cbd3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68c00017f0 con 0x7f68c40a98f0 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68c8ad7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68c8ad7640 1 -- 172.21.15.134:0/1175005446 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f68c0001c00 con 0x7f68c40a98f0 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68c8ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68c8ad7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68c8ad7640 10 monclient: dump: 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:52.168 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68cbd3d640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68cbd3d640 4 set_mon_vals no callback set 2024-03-25T19:35:52.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68cbd3d640 10 monclient: shutdown 2024-03-25T19:35:52.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68cbd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:52.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.167+0000 7f68cbd3d640 1 -- 172.21.15.134:0/1175005446 >> v1:172.21.15.159:6789/0 conn(0x7f68c40a98f0 legacy=0x7f68c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.168+0000 7f68cbd3d640 1 -- 172.21.15.134:0/1175005446 shutdown_connections 2024-03-25T19:35:52.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.168+0000 7f68cbd3d640 1 -- 172.21.15.134:0/1175005446 >> 172.21.15.134:0/1175005446 conn(0x7f68c40b7ea0 msgr2=0x7f68c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:52.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.168+0000 7f68cbd3d640 1 -- 172.21.15.134:0/1175005446 shutdown_connections 2024-03-25T19:35:52.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.168+0000 7f68cbd3d640 1 -- 172.21.15.134:0/1175005446 wait complete. 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.168+0000 7f68cbd3d640 10 monclient: build_initial_monmap 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.168+0000 7f68cbd3d640 10 monclient: monmap: 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 librados: starting msgr at 2024-03-25T19:35:52.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 librados: starting objecter 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 Processor -- start 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 -- start start 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 librados: setting wanted keys 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 librados: calling monclient init 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient: init 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f68c40a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient(hunting): picked mon.noname-a con 0x7f68c413f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:52.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f68c41430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4139610 con 0x7f68c413f710 2024-03-25T19:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4139900 con 0x7f68c40a98f0 2024-03-25T19:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c4139a90 con 0x7f68c41430c0 2024-03-25T19:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:52.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.169+0000 7f68cbd3d640 10 monclient(hunting): authenticate will time out at 6384.328622s 2024-03-25T19:35:52.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.177+0000 7f68ca2da640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f68c41430c0 0x7f68c4133260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:49496/0 (socket says 172.21.15.134:49496) 2024-03-25T19:35:52.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.178+0000 7f68ca2da640 1 -- 172.21.15.134:0/3889891912 learned_addr learned my addr 172.21.15.134:0/3889891912 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:52.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.181+0000 7f68ba7fc640 10 client.?.objecter ms_handle_connect 0x7f68c41430c0 2024-03-25T19:35:52.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.181+0000 7f68ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:52.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.183+0000 7f68ba7fc640 10 client.?.objecter ms_handle_connect 0x7f68c413f710 2024-03-25T19:35:52.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.183+0000 7f68ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:52.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.187+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309045068 0 0) 0x7f68c4139a90 con 0x7f68c41430c0 2024-03-25T19:35:52.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.187+0000 7f68ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.187+0000 7f68ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.187+0000 7f68ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.187+0000 7f68ba7fc640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:35:52.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.187+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a8004eb0 con 0x7f68c41430c0 2024-03-25T19:35:52.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.191+0000 7f68ba7fc640 10 client.?.objecter ms_handle_connect 0x7f68c40a98f0 2024-03-25T19:35:52.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.191+0000 7f68ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:52.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.202+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363511108 0 0) 0x7f68c4139610 con 0x7f68c413f710 2024-03-25T19:35:52.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.202+0000 7f68ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.202+0000 7f68ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.202+0000 7f68ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.202+0000 7f68ba7fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:35:52.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.202+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c4139a90 con 0x7f68c413f710 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.204+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1365128123 0 0) 0x7f68a8004eb0 con 0x7f68c41430c0 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.204+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68c4139610 con 0x7f68c41430c0 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.204+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f68bc001c90 con 0x7f68c41430c0 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.204+0000 7f68ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.204+0000 7f68ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.204+0000 7f68ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.204 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.205 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.205 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:52.205 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:52.205 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:52.205 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:52.205 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:52.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.204+0000 7f68ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:52.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.208+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929078318 0 0) 0x7f68c4139900 con 0x7f68c40a98f0 2024-03-25T19:35:52.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.208+0000 7f68ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.208+0000 7f68ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.208+0000 7f68ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.208+0000 7f68ba7fc640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:35:52.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.208+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a8004eb0 con 0x7f68c40a98f0 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.211+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1956292787 0 0) 0x7f68c4139a90 con 0x7f68c413f710 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.211+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68c4139900 con 0x7f68c413f710 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.211+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f68b4001930 con 0x7f68c413f710 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.211+0000 7f68ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.211+0000 7f68ba7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.211+0000 7f68ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:52.211 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:52.212 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2846882791 0 0) 0x7f68c4139610 con 0x7f68c41430c0 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 >> v1:172.21.15.159:6789/0 conn(0x7f68c40a98f0 legacy=0x7f68c41320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 >> v1:172.21.15.134:6789/0 conn(0x7f68c413f710 legacy=0x7f68c4132800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 1 monclient: found mon.c 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c4138bf0 con 0x7f68c41430c0 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:52.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.221+0000 7f68cbd3d640 5 monclient: authenticate success, global_id 4547 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 10 monclient: _renew_subs 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c4138e40 con 0x7f68c41430c0 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 10 monclient: _renew_subs 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c4138380 con 0x7f68c41430c0 2024-03-25T19:35:52.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.222+0000 7f68cbd3d640 1 librados: init done 2024-03-25T19:35:52.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.223+0000 7f68cbd3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:52.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.223+0000 7f68cbd3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:52.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.223+0000 7f68cbd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:52.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.223+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c4138680 con 0x7f68c41430c0 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.437+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68bc003970 con 0x7f68c41430c0 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.437+0000 7f68ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.437+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f68bc003d80 con 0x7f68c41430c0 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.437+0000 7f68ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.437+0000 7f68ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.437+0000 7f68ba7fc640 10 monclient: dump: 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:52.437 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:52.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.437+0000 7f68bbfff640 4 set_mon_vals no callback set 2024-03-25T19:35:52.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.514+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f68bc003ee0 con 0x7f68c41430c0 2024-03-25T19:35:52.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.514+0000 7f68c9ad9640 10 monclient: get_auth_request con 0x7f68a806df20 auth_method 0 2024-03-25T19:35:52.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.515+0000 7f68ba7fc640 10 client.4547.objecter ms_handle_connect 0x7f68a806df20 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.520+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f68bc054a80 con 0x7f68c41430c0 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.520+0000 7f68ba7fc640 10 client.4547.objecter ms_dispatch 0x7f68c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.520+0000 7f68ba7fc640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.520+0000 7f68ba7fc640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.520+0000 7f68ba7fc640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.520+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68bc050590 con 0x7f68c41430c0 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.521+0000 7f68ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:52.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.521+0000 7f68ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:52.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.744+0000 7f68cbd3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:52.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.744+0000 7f68cbd3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:52.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.744+0000 7f68cbd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:52.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.744+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68c40627b0 con 0x7f68c41430c0 2024-03-25T19:35:52.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.808+0000 7f68ba7fc640 1 -- 172.21.15.134:0/3889891912 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f68bc0524c0 con 0x7f68c41430c0 2024-03-25T19:35:52.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.808+0000 7f68ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:52.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.808+0000 7f68ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:52.809 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:35:52.809 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:52.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 10 librados: watch_flush enter 2024-03-25T19:35:52.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 10 librados: watch_flush exit 2024-03-25T19:35:52.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T19:35:52.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 10 client.4547.objecter successfully canceled tick 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 >> v1:172.21.15.159:6800/1875874887 conn(0x7f68a806df20 legacy=0x7f68a80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 10 monclient: shutdown 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 >> v1:172.21.15.134:6790/0 conn(0x7f68c41430c0 legacy=0x7f68c4133260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 shutdown_connections 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 >> 172.21.15.134:0/3889891912 conn(0x7f68c40b7ea0 msgr2=0x7f68c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 shutdown_connections 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 1 -- 172.21.15.134:0/3889891912 wait complete. 2024-03-25T19:35:52.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:52.811+0000 7f68cbd3d640 1 librados: shutdown 2024-03-25T19:35:52.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:59.630 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.751+0000 7fb773e8f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.751+0000 7fb773e8f640 10 monclient: build_initial_monmap 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.751+0000 7fb773e8f640 10 monclient: monmap: 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:35:59.752391+0000 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:35:59.752391+0000 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:59.751 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:59.752 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:59.752 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:35:59.752 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:35:59.752 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:59.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.752+0000 7fb773e8f640 1 Processor -- start 2024-03-25T19:35:59.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.752+0000 7fb773e8f640 1 -- start start 2024-03-25T19:35:59.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.752+0000 7fb773e8f640 10 monclient: init 2024-03-25T19:35:59.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:59.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 10 monclient(hunting): picked mon.noname-c con 0x7fb76c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 10 monclient(hunting): picked mon.noname-a con 0x7fb76c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 10 monclient(hunting): picked mon.noname-b con 0x7fb76c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c142520 con 0x7fb76c13f710 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0b28a0 con 0x7fb76c0a98f0 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0b2a30 con 0x7fb76c1430c0 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:59.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.753+0000 7fb773e8f640 10 monclient(hunting): authenticate will time out at 6391.912127s 2024-03-25T19:35:59.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.763+0000 7fb77242c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fb76c1430c0 0x7fb76c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:53998/0 (socket says 172.21.15.134:53998) 2024-03-25T19:35:59.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.763+0000 7fb77242c640 1 -- 172.21.15.134:0/2789115347 learned_addr learned my addr 172.21.15.134:0/2789115347 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:59.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.785+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514813855 0 0) 0x7fb76c0b28a0 con 0x7fb76c0a98f0 2024-03-25T19:35:59.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.785+0000 7fb770c29640 10 monclient(hunting): _negotiate 2024-03-25T19:35:59.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.785+0000 7fb770c29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:59.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.785+0000 7fb770c29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:59.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.785+0000 7fb770c29640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:35:59.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.785+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb758003520 con 0x7fb76c0a98f0 2024-03-25T19:35:59.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.787+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776520110 0 0) 0x7fb76c0b2a30 con 0x7fb76c1430c0 2024-03-25T19:35:59.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.787+0000 7fb770c29640 10 monclient(hunting): _negotiate 2024-03-25T19:35:59.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.787+0000 7fb770c29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:59.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.787+0000 7fb770c29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:59.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.787+0000 7fb770c29640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:35:59.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.787+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb76c0b28a0 con 0x7fb76c1430c0 2024-03-25T19:35:59.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.791+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787763710 0 0) 0x7fb76c142520 con 0x7fb76c13f710 2024-03-25T19:35:59.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.791+0000 7fb770c29640 10 monclient(hunting): _negotiate 2024-03-25T19:35:59.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.791+0000 7fb770c29640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:59.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.791+0000 7fb770c29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:59.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.791+0000 7fb770c29640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:35:59.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.791+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb76c0b2a30 con 0x7fb76c13f710 2024-03-25T19:35:59.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340276219 0 0) 0x7fb758003520 con 0x7fb76c0a98f0 2024-03-25T19:35:59.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 >> v1:172.21.15.134:6790/0 conn(0x7fb76c1430c0 legacy=0x7fb76c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 >> v1:172.21.15.134:6789/0 conn(0x7fb76c13f710 legacy=0x7fb76c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 10 monclient: _finish_hunting 0 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 1 monclient: found mon.noname-c 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c0b44c0 con 0x7fb76c0a98f0 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 10 monclient: _finish_auth 0 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 10 monclient: _check_auth_tickets 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb768001690 con 0x7fb76c0a98f0 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb773e8f640 5 monclient: authenticate success, global_id 4543 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb773e8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:59.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb770c29640 10 monclient: dump: 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:59.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.802+0000 7fb773e8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7680017f0 con 0x7fb76c0a98f0 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb770c29640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb770c29640 1 -- 172.21.15.134:0/2789115347 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb768001c00 con 0x7fb76c0a98f0 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb770c29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb770c29640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb770c29640 10 monclient: dump: 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.831 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:59.832 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:59.832 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:59.832 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:59.832 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:59.832 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:59.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb773e8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:59.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.831+0000 7fb773e8f640 4 set_mon_vals no callback set 2024-03-25T19:35:59.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.832+0000 7fb773e8f640 10 monclient: shutdown 2024-03-25T19:35:59.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.832+0000 7fb773e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:59.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.832+0000 7fb773e8f640 1 -- 172.21.15.134:0/2789115347 >> v1:172.21.15.159:6789/0 conn(0x7fb76c0a98f0 legacy=0x7fb76c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:59.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.832+0000 7fb773e8f640 1 -- 172.21.15.134:0/2789115347 shutdown_connections 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.832+0000 7fb773e8f640 1 -- 172.21.15.134:0/2789115347 >> 172.21.15.134:0/2789115347 conn(0x7fb76c0b7ea0 msgr2=0x7fb76c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.832+0000 7fb773e8f640 1 -- 172.21.15.134:0/2789115347 shutdown_connections 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.832+0000 7fb773e8f640 1 -- 172.21.15.134:0/2789115347 wait complete. 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 10 monclient: build_initial_monmap 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 10 monclient: monmap: 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:35:59.834 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 1 librados: starting msgr at 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 1 librados: starting objecter 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 1 Processor -- start 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 1 -- start start 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 1 librados: setting wanted keys 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 1 librados: calling monclient init 2024-03-25T19:35:59.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.833+0000 7fb773e8f640 10 monclient: init 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 10 monclient(hunting): picked mon.noname-b con 0x7fb76c13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 10 monclient(hunting): picked mon.noname-c con 0x7fb76c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 10 monclient(hunting): picked mon.noname-a con 0x7fb76c1387a0 addr v1:172.21.15.134:6789/0 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0abb50 con 0x7fb76c1387a0 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0abe30 con 0x7fb76c13f710 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0ac110 con 0x7fb76c1430c0 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:59.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.834+0000 7fb773e8f640 10 monclient(hunting): authenticate will time out at 6391.993211s 2024-03-25T19:35:59.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.846+0000 7fb77242c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb76c1387a0 0x7fb76c1e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58400/0 (socket says 172.21.15.134:58400) 2024-03-25T19:35:59.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.846+0000 7fb77242c640 1 -- 172.21.15.134:0/2308474206 learned_addr learned my addr 172.21.15.134:0/2308474206 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:35:59.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.849+0000 7fb7627fc640 10 client.?.objecter ms_handle_connect 0x7fb76c1430c0 2024-03-25T19:35:59.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.849+0000 7fb7627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:59.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.851+0000 7fb7627fc640 10 client.?.objecter ms_handle_connect 0x7fb76c13f710 2024-03-25T19:35:59.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.851+0000 7fb7627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:59.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.851+0000 7fb7627fc640 10 client.?.objecter ms_handle_connect 0x7fb76c1387a0 2024-03-25T19:35:59.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.851+0000 7fb7627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:59.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.867+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818470 0 0) 0x7fb76c0abb50 con 0x7fb76c1387a0 2024-03-25T19:35:59.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.867+0000 7fb7627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:59.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.867+0000 7fb7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:59.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.867+0000 7fb7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:59.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.867+0000 7fb7627fc640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:35:59.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.867+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb750004eb0 con 0x7fb76c1387a0 2024-03-25T19:35:59.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.869+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480810932 0 0) 0x7fb76c0abe30 con 0x7fb76c13f710 2024-03-25T19:35:59.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.869+0000 7fb7627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:59.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.869+0000 7fb7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:59.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.869+0000 7fb7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.869+0000 7fb7627fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.869+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb76c0abb50 con 0x7fb76c13f710 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.870+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408820430 0 0) 0x7fb76c0ac110 con 0x7fb76c1430c0 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.870+0000 7fb7627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.870+0000 7fb7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.870+0000 7fb7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.870+0000 7fb7627fc640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:35:59.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.870+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb76c0abe30 con 0x7fb76c1430c0 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.883+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456305183 0 0) 0x7fb750004eb0 con 0x7fb76c1387a0 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.883+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb76c0ac110 con 0x7fb76c1387a0 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.883+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb764001c70 con 0x7fb76c1387a0 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.883+0000 7fb7627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.883+0000 7fb7627fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.883+0000 7fb7627fc640 10 monclient(hunting): dump: 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:59.883 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:59.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.883+0000 7fb7627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:59.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.889+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 911218679 0 0) 0x7fb76c0abe30 con 0x7fb76c1430c0 2024-03-25T19:35:59.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.889+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb750004eb0 con 0x7fb76c1430c0 2024-03-25T19:35:59.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.889+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb75c001830 con 0x7fb76c1430c0 2024-03-25T19:35:59.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.889+0000 7fb7627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:59.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.889+0000 7fb7627fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:59.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.889+0000 7fb7627fc640 10 monclient(hunting): dump: 2024-03-25T19:35:59.889 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:59.890 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.909+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2861335783 0 0) 0x7fb76c0abb50 con 0x7fb76c13f710 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.909+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb76c0abe30 con 0x7fb76c13f710 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.909+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb768001b00 con 0x7fb76c13f710 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.909+0000 7fb7627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.909+0000 7fb7627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.909+0000 7fb7627fc640 10 monclient(hunting): dump: 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:35:59.909 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:35:59.910 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:35:59.910 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 626090495 0 0) 0x7fb750004eb0 con 0x7fb76c1430c0 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 >> v1:172.21.15.159:6789/0 conn(0x7fb76c13f710 legacy=0x7fb76c0ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 >> v1:172.21.15.134:6789/0 conn(0x7fb76c1387a0 legacy=0x7fb76c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 1 monclient: found mon.c 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c139110 con 0x7fb76c1430c0 2024-03-25T19:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb7627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 5 monclient: authenticate success, global_id 4553 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 10 monclient: _renew_subs 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c137e20 con 0x7fb76c1430c0 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 10 monclient: _renew_subs 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c138dd0 con 0x7fb76c1430c0 2024-03-25T19:35:59.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.917+0000 7fb773e8f640 1 librados: init done 2024-03-25T19:35:59.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.918+0000 7fb773e8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:59.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.918+0000 7fb773e8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:59.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.918+0000 7fb773e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:35:59.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:35:59.918+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76c0b3ef0 con 0x7fb76c1430c0 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.244+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb75c001d40 con 0x7fb76c1430c0 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.244+0000 7fb7627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.244+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb75c0037b0 con 0x7fb76c1430c0 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.244+0000 7fb7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.244+0000 7fb7627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.244+0000 7fb7627fc640 10 monclient: dump: 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:00.244 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:00.245 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:00.245 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:00.245 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:00.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.244+0000 7fb763fff640 4 set_mon_vals no callback set 2024-03-25T19:36:00.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.245+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fb75c019840 con 0x7fb76c1430c0 2024-03-25T19:36:00.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.245+0000 7fb771c2b640 10 monclient: get_auth_request con 0x7fb75006e180 auth_method 0 2024-03-25T19:36:00.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.246+0000 7fb7627fc640 10 client.4553.objecter ms_handle_connect 0x7fb75006e180 2024-03-25T19:36:00.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fb75c053520 con 0x7fb76c1430c0 2024-03-25T19:36:00.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 10 client.4553.objecter ms_dispatch 0x7fb76c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:00.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:00.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:00.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T19:36:00.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb75c04ee60 con 0x7fb76c1430c0 2024-03-25T19:36:00.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:00.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.251+0000 7fb7627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:00.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.490+0000 7fb773e8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:00.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.490+0000 7fb773e8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:00.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.490+0000 7fb773e8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:00.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.490+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb76c0627b0 con 0x7fb76c1430c0 2024-03-25T19:36:00.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.591+0000 7fb7627fc640 1 -- 172.21.15.134:0/2308474206 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fb75c050f60 con 0x7fb76c1430c0 2024-03-25T19:36:00.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.591+0000 7fb7627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:00.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.591+0000 7fb7627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:00.591 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:00.592 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 10 librados: watch_flush enter 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 10 librados: watch_flush exit 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 10 client.4553.objecter successfully canceled tick 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 >> v1:172.21.15.159:6800/1875874887 conn(0x7fb75006e180 legacy=0x7fb750070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 10 monclient: shutdown 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:00.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.593+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 >> v1:172.21.15.134:6790/0 conn(0x7fb76c1430c0 legacy=0x7fb76c136500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:00.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.594+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 shutdown_connections 2024-03-25T19:36:00.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.594+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 >> 172.21.15.134:0/2308474206 conn(0x7fb76c0b7ea0 msgr2=0x7fb76c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:00.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.594+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 shutdown_connections 2024-03-25T19:36:00.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.594+0000 7fb773e8f640 1 -- 172.21.15.134:0/2308474206 wait complete. 2024-03-25T19:36:00.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:00.594+0000 7fb773e8f640 1 librados: shutdown 2024-03-25T19:36:00.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:05.956 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:10.628 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:05.962 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:10.609 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:05.978 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:10.650 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:06.003 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:10.675 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:06.022 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:10.669 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:06.033 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:10.706 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:06.037 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:10.684 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:06.046 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:10.719 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:06.099 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:10.771 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:07.412 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.530+0000 7f892aa4a640 10 monclient: get_monmap_and_config 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.530+0000 7f892aa4a640 10 monclient: build_initial_monmap 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.530+0000 7f892aa4a640 10 monclient: monmap: 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:36:07.531254+0000 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:36:07.531254+0000 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:07.530 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:36:07.531 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:36:07.531 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.530+0000 7f892aa4a640 1 Processor -- start 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 1 -- start start 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 10 monclient: init 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f89240b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f89240b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f892414be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89240af940 con 0x7f89240b4d60 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.531+0000 7f892aa4a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89240b0a70 con 0x7f892414be90 2024-03-25T19:36:07.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.532+0000 7f892aa4a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89240b0d50 con 0x7f89240b1730 2024-03-25T19:36:07.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.532+0000 7f892aa4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:07.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.532+0000 7f892aa4a640 10 monclient(hunting): authenticate will time out at 6399.690660s 2024-03-25T19:36:07.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.541+0000 7f8928fe7640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f892414be90 0x7f892414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:59750/0 (socket says 172.21.15.134:59750) 2024-03-25T19:36:07.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.541+0000 7f8928fe7640 1 -- 172.21.15.134:0/1524675437 learned_addr learned my addr 172.21.15.134:0/1524675437 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:07.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.560+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511114352 0 0) 0x7f89240b0a70 con 0x7f892414be90 2024-03-25T19:36:07.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.560+0000 7f8922ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:07.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.560+0000 7f8922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:07.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.560+0000 7f8922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:07.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.560+0000 7f8922ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:36:07.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.560+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f890c003520 con 0x7f892414be90 2024-03-25T19:36:07.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.563+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307172373 0 0) 0x7f89240b0d50 con 0x7f89240b1730 2024-03-25T19:36:07.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.563+0000 7f8922ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:07.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.563+0000 7f8922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:07.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.563+0000 7f8922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:07.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.563+0000 7f8922ffd640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:36:07.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.563+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89240b0a70 con 0x7f89240b1730 2024-03-25T19:36:07.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.570+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554583644 0 0) 0x7f89240af940 con 0x7f89240b4d60 2024-03-25T19:36:07.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.570+0000 7f8922ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:07.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.570+0000 7f8922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:07.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.570+0000 7f8922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:07.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.570+0000 7f8922ffd640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:36:07.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.570+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89240b0d50 con 0x7f89240b4d60 2024-03-25T19:36:07.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591040595 0 0) 0x7f89240b0a70 con 0x7f89240b1730 2024-03-25T19:36:07.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 >> v1:172.21.15.159:6789/0 conn(0x7f892414be90 legacy=0x7f892414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:07.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 >> v1:172.21.15.134:6789/0 conn(0x7f89240b4d60 legacy=0x7f892413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:07.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:07.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 1 monclient: found mon.noname-b 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240b1030 con 0x7f89240b1730 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8918001670 con 0x7f89240b1730 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f8922ffd640 10 monclient: dump: 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.578 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f892aa4a640 5 monclient: authenticate success, global_id 4556 2024-03-25T19:36:07.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.577+0000 7f892aa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89180017d0 con 0x7f89240b1730 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f8922ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f8922ffd640 1 -- 172.21.15.134:0/1524675437 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8918001be0 con 0x7f89240b1730 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f8922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f8922ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f8922ffd640 10 monclient: dump: 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:07.597 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:07.598 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:07.598 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:07.598 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:07.598 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:07.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f892aa4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:07.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.597+0000 7f892aa4a640 4 set_mon_vals no callback set 2024-03-25T19:36:07.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.598+0000 7f892aa4a640 10 monclient: shutdown 2024-03-25T19:36:07.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.598+0000 7f892aa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:07.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.598+0000 7f892aa4a640 1 -- 172.21.15.134:0/1524675437 >> v1:172.21.15.134:6790/0 conn(0x7f89240b1730 legacy=0x7f89240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.598+0000 7f892aa4a640 1 -- 172.21.15.134:0/1524675437 shutdown_connections 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.598+0000 7f892aa4a640 1 -- 172.21.15.134:0/1524675437 >> 172.21.15.134:0/1524675437 conn(0x7f89240617a0 msgr2=0x7f8924061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.598+0000 7f892aa4a640 1 -- 172.21.15.134:0/1524675437 shutdown_connections 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.598+0000 7f892aa4a640 1 -- 172.21.15.134:0/1524675437 wait complete. 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 10 monclient: build_initial_monmap 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 10 monclient: monmap: 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:36:07.600 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 1 librados: starting msgr at 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 1 librados: starting objecter 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 1 Processor -- start 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 1 -- start start 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 1 librados: setting wanted keys 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 1 librados: calling monclient init 2024-03-25T19:36:07.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.599+0000 7f892aa4a640 10 monclient: init 2024-03-25T19:36:07.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.600+0000 7f892aa4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:07.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.600+0000 7f892aa4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:07.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.600+0000 7f892aa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f89240b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:07.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.600+0000 7f892aa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f89240b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:07.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.601+0000 7f892aa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f892414be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:07.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.601+0000 7f892aa4a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89241381d0 con 0x7f892414be90 2024-03-25T19:36:07.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.601+0000 7f892aa4a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89241397e0 con 0x7f89240b1730 2024-03-25T19:36:07.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.601+0000 7f892aa4a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8924137cd0 con 0x7f89240b4d60 2024-03-25T19:36:07.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.601+0000 7f892aa4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:07.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.601+0000 7f892aa4a640 10 monclient(hunting): authenticate will time out at 6399.759711s 2024-03-25T19:36:07.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.614+0000 7f89237fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f89240b4d60 0x7f89241366e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:45776/0 (socket says 172.21.15.134:45776) 2024-03-25T19:36:07.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.614+0000 7f89237fe640 1 -- 172.21.15.134:0/3233207453 learned_addr learned my addr 172.21.15.134:0/3233207453 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:07.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.615+0000 7f8920ff9640 10 client.?.objecter ms_handle_connect 0x7f892414be90 2024-03-25T19:36:07.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.616+0000 7f8920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:07.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.616+0000 7f8920ff9640 10 client.?.objecter ms_handle_connect 0x7f89240b4d60 2024-03-25T19:36:07.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.616+0000 7f8920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.625+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376101765 0 0) 0x7f89241381d0 con 0x7f892414be90 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.625+0000 7f8920ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.625+0000 7f8920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.625+0000 7f8920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.625+0000 7f8920ff9640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.625+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910004eb0 con 0x7f892414be90 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.626+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543341648 0 0) 0x7f8924137cd0 con 0x7f89240b4d60 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.626+0000 7f8920ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.626+0000 7f8920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.626+0000 7f8920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.626+0000 7f8920ff9640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:36:07.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.626+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89241381d0 con 0x7f89240b4d60 2024-03-25T19:36:07.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.628+0000 7f8920ff9640 10 client.?.objecter ms_handle_connect 0x7f89240b1730 2024-03-25T19:36:07.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.628+0000 7f8920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:07.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.640+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420172832 0 0) 0x7f89241397e0 con 0x7f89240b1730 2024-03-25T19:36:07.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.640+0000 7f8920ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:07.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.640+0000 7f8920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:07.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.640+0000 7f8920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:07.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.640+0000 7f8920ff9640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:36:07.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.640+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8924137cd0 con 0x7f89240b1730 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.642+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2617425548 0 0) 0x7f89241381d0 con 0x7f89240b4d60 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.642+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89241397e0 con 0x7f89240b4d60 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.642+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8918001670 con 0x7f89240b4d60 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.642+0000 7f8920ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.642+0000 7f8920ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.642+0000 7f8920ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.642 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.643 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:07.643 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:07.643 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:07.643 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:07.643 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:07.643 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:07.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.642+0000 7f8920ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.645+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 266864122 0 0) 0x7f8910004eb0 con 0x7f892414be90 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.645+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89241381d0 con 0x7f892414be90 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.645+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f891c001af0 con 0x7f892414be90 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.645+0000 7f8920ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.645+0000 7f8920ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.645+0000 7f8920ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:07.645 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:07.646 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301377352 0 0) 0x7f89241397e0 con 0x7f89240b4d60 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 >> v1:172.21.15.159:6789/0 conn(0x7f89240b1730 legacy=0x7f8924135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 >> v1:172.21.15.134:6789/0 conn(0x7f892414be90 legacy=0x7f89241e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 1 monclient: found mon.c 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8924137e60 con 0x7f89240b4d60 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f8920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f892aa4a640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f892aa4a640 10 monclient: _renew_subs 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f892aa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:07.656 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8924138df0 con 0x7f89240b4d60 2024-03-25T19:36:07.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f892aa4a640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:07.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f892aa4a640 10 monclient: _renew_subs 2024-03-25T19:36:07.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.655+0000 7f892aa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:07.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.656+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8924145130 con 0x7f89240b4d60 2024-03-25T19:36:07.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.656+0000 7f892aa4a640 1 librados: init done 2024-03-25T19:36:07.657 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.657+0000 7f892aa4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:07.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.657+0000 7f892aa4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:07.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.657+0000 7f892aa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:07.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.657+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89241401f0 con 0x7f89240b4d60 2024-03-25T19:36:07.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.939+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89180038a0 con 0x7f89240b4d60 2024-03-25T19:36:07.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.939+0000 7f8920ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:07.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.939+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8918003cf0 con 0x7f89240b4d60 2024-03-25T19:36:07.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.939+0000 7f8920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:07.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.939+0000 7f8920ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:07.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.939+0000 7f8920ff9640 10 monclient: dump: 2024-03-25T19:36:07.939 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:07.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.939+0000 7f89227fc640 4 set_mon_vals no callback set 2024-03-25T19:36:07.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.940+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f8918003f10 con 0x7f89240b4d60 2024-03-25T19:36:07.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.940+0000 7f8923fff640 10 monclient: get_auth_request con 0x7f891006dec0 auth_method 0 2024-03-25T19:36:07.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.941+0000 7f8920ff9640 10 client.4559.objecter ms_handle_connect 0x7f891006dec0 2024-03-25T19:36:07.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f8918054910 con 0x7f89240b4d60 2024-03-25T19:36:07.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 10 client.4559.objecter ms_dispatch 0x7f8924000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:07.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:07.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:07.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T19:36:07.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8918050420 con 0x7f89240b4d60 2024-03-25T19:36:07.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:07.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:07.946+0000 7f8920ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:08.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.192+0000 7f892aa4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:08.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.192+0000 7f892aa4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:08.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.192+0000 7f892aa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:08.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.192+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89240627b0 con 0x7f89240b4d60 2024-03-25T19:36:08.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.218+0000 7f8920ff9640 1 -- 172.21.15.134:0/3233207453 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f8918052350 con 0x7f89240b4d60 2024-03-25T19:36:08.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.218+0000 7f8920ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:08.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.218+0000 7f8920ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:08.218 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:08.218 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 10 librados: watch_flush enter 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 10 librados: watch_flush exit 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 10 client.4559.objecter successfully canceled tick 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 >> v1:172.21.15.159:6800/1875874887 conn(0x7f891006dec0 legacy=0x7f8910070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 10 monclient: shutdown 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.220+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 >> v1:172.21.15.134:6790/0 conn(0x7f89240b4d60 legacy=0x7f89241366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.221+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 shutdown_connections 2024-03-25T19:36:08.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.221+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 >> 172.21.15.134:0/3233207453 conn(0x7f89240617a0 msgr2=0x7f89240b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:08.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.221+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 shutdown_connections 2024-03-25T19:36:08.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.221+0000 7f892aa4a640 1 -- 172.21.15.134:0/3233207453 wait complete. 2024-03-25T19:36:08.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:08.221+0000 7f892aa4a640 1 librados: shutdown 2024-03-25T19:36:08.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:15.039 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.158+0000 7fe0906ed640 10 monclient: get_monmap_and_config 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.158+0000 7fe0906ed640 10 monclient: build_initial_monmap 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.158+0000 7fe0906ed640 10 monclient: monmap: 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:36:15.159477+0000 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:36:15.159477+0000 2024-03-25T19:36:15.158 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.159+0000 7fe0906ed640 1 Processor -- start 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.159+0000 7fe0906ed640 1 -- start start 2024-03-25T19:36:15.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.159+0000 7fe0906ed640 10 monclient: init 2024-03-25T19:36:15.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:15.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:15.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 10 monclient(hunting): picked mon.noname-b con 0x7fe0880a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:15.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 10 monclient(hunting): picked mon.noname-a con 0x7fe08813f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:15.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 10 monclient(hunting): picked mon.noname-c con 0x7fe0881430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:15.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe088142520 con 0x7fe08813f710 2024-03-25T19:36:15.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0880b28a0 con 0x7fe0881430c0 2024-03-25T19:36:15.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0880b2a30 con 0x7fe0880a98f0 2024-03-25T19:36:15.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:15.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.160+0000 7fe0906ed640 10 monclient(hunting): authenticate will time out at 6407.319412s 2024-03-25T19:36:15.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.168+0000 7fe08e489640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe0880a98f0 0x7fe0880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:34042/0 (socket says 172.21.15.134:34042) 2024-03-25T19:36:15.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.168+0000 7fe08e489640 1 -- 172.21.15.134:0/1182760614 learned_addr learned my addr 172.21.15.134:0/1182760614 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:15.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.186+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682573946 0 0) 0x7fe088142520 con 0x7fe08813f710 2024-03-25T19:36:15.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.186+0000 7fe08d487640 10 monclient(hunting): _negotiate 2024-03-25T19:36:15.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.186+0000 7fe08d487640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:15.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.186+0000 7fe08d487640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:15.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.186+0000 7fe08d487640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:36:15.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.186+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe074003520 con 0x7fe08813f710 2024-03-25T19:36:15.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.189+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631861723 0 0) 0x7fe0880b2a30 con 0x7fe0880a98f0 2024-03-25T19:36:15.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.189+0000 7fe08d487640 10 monclient(hunting): _negotiate 2024-03-25T19:36:15.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.189+0000 7fe08d487640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:15.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.189+0000 7fe08d487640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:15.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.189+0000 7fe08d487640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:36:15.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.189+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe088142520 con 0x7fe0880a98f0 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009823623 0 0) 0x7fe088142520 con 0x7fe0880a98f0 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 >> v1:172.21.15.159:6789/0 conn(0x7fe0881430c0 legacy=0x7fe08814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 >> v1:172.21.15.134:6789/0 conn(0x7fe08813f710 legacy=0x7fe0881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 10 monclient: _finish_hunting 0 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 1 monclient: found mon.noname-b 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880b44c0 con 0x7fe0880a98f0 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 10 monclient: _finish_auth 0 2024-03-25T19:36:15.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 10 monclient: _check_auth_tickets 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe084001670 con 0x7fe0880a98f0 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe0906ed640 5 monclient: authenticate success, global_id 4562 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe0906ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe08d487640 10 monclient: dump: 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:15.199 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:15.200 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:15.200 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:15.200 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:15.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.197+0000 7fe0906ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0840017d0 con 0x7fe0880a98f0 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe08d487640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe08d487640 1 -- 172.21.15.134:0/1182760614 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe084001ba0 con 0x7fe0880a98f0 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe08d487640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe08d487640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe08d487640 10 monclient: dump: 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:15.219 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:15.220 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:15.220 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:15.220 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:15.220 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:15.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe0906ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:15.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.219+0000 7fe0906ed640 4 set_mon_vals no callback set 2024-03-25T19:36:15.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.220+0000 7fe0906ed640 10 monclient: shutdown 2024-03-25T19:36:15.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.220+0000 7fe0906ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:15.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.220+0000 7fe0906ed640 1 -- 172.21.15.134:0/1182760614 >> v1:172.21.15.134:6790/0 conn(0x7fe0880a98f0 legacy=0x7fe0880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:15.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.220+0000 7fe0906ed640 1 -- 172.21.15.134:0/1182760614 shutdown_connections 2024-03-25T19:36:15.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.220+0000 7fe0906ed640 1 -- 172.21.15.134:0/1182760614 >> 172.21.15.134:0/1182760614 conn(0x7fe0880b7ea0 msgr2=0x7fe0880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.220+0000 7fe0906ed640 1 -- 172.21.15.134:0/1182760614 shutdown_connections 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.220+0000 7fe0906ed640 1 -- 172.21.15.134:0/1182760614 wait complete. 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 10 monclient: build_initial_monmap 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 10 monclient: monmap: 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 1 librados: starting msgr at 2024-03-25T19:36:15.222 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 1 librados: starting objecter 2024-03-25T19:36:15.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 1 Processor -- start 2024-03-25T19:36:15.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 1 -- start start 2024-03-25T19:36:15.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 1 librados: setting wanted keys 2024-03-25T19:36:15.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 1 librados: calling monclient init 2024-03-25T19:36:15.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.221+0000 7fe0906ed640 10 monclient: init 2024-03-25T19:36:15.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:15.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:15.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 10 monclient(hunting): picked mon.noname-a con 0x7fe0880a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:15.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 10 monclient(hunting): picked mon.noname-b con 0x7fe08813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:15.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 10 monclient(hunting): picked mon.noname-c con 0x7fe0881430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:15.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0880ab9c0 con 0x7fe0880a98f0 2024-03-25T19:36:15.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0880abca0 con 0x7fe08813f710 2024-03-25T19:36:15.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe088137e80 con 0x7fe0881430c0 2024-03-25T19:36:15.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:15.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.222+0000 7fe0906ed640 10 monclient(hunting): authenticate will time out at 6407.381590s 2024-03-25T19:36:15.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.224+0000 7fe08ec8a640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe0881430c0 0x7fe0881e2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:34088/0 (socket says 172.21.15.134:34088) 2024-03-25T19:36:15.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.224+0000 7fe08ec8a640 1 -- 172.21.15.134:0/1142092102 learned_addr learned my addr 172.21.15.134:0/1142092102 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:15.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.227+0000 7fe07effd640 10 client.?.objecter ms_handle_connect 0x7fe0881430c0 2024-03-25T19:36:15.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.227+0000 7fe07effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:15.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.236+0000 7fe07effd640 10 client.?.objecter ms_handle_connect 0x7fe0880a98f0 2024-03-25T19:36:15.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.236+0000 7fe07effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:15.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.241+0000 7fe07effd640 10 client.?.objecter ms_handle_connect 0x7fe08813f710 2024-03-25T19:36:15.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.241+0000 7fe07effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:15.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.245+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780625198 0 0) 0x7fe088137e80 con 0x7fe0881430c0 2024-03-25T19:36:15.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.245+0000 7fe07effd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:15.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.245+0000 7fe07effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:15.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.245+0000 7fe07effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:15.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.245+0000 7fe07effd640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:36:15.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.245+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe06c004eb0 con 0x7fe0881430c0 2024-03-25T19:36:15.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.257+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532235208 0 0) 0x7fe0880ab9c0 con 0x7fe0880a98f0 2024-03-25T19:36:15.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.257+0000 7fe07effd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:15.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.257+0000 7fe07effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:15.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.257+0000 7fe07effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:15.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.257+0000 7fe07effd640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:36:15.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.257+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe088137e80 con 0x7fe0880a98f0 2024-03-25T19:36:15.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.272+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242502575 0 0) 0x7fe0880abca0 con 0x7fe08813f710 2024-03-25T19:36:15.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.272+0000 7fe07effd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:15.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.272+0000 7fe07effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:15.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.272+0000 7fe07effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:15.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.272+0000 7fe07effd640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:36:15.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.272+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0880ab9c0 con 0x7fe08813f710 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.275+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1548058816 0 0) 0x7fe088137e80 con 0x7fe0880a98f0 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.275+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0880abca0 con 0x7fe0880a98f0 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.275+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe084001ae0 con 0x7fe0880a98f0 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.275+0000 7fe07effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.275+0000 7fe07effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.275+0000 7fe07effd640 10 monclient(hunting): dump: 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.275 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:15.276 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:15.276 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:15.276 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:15.276 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:15.276 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:15.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.275+0000 7fe07effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.345+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1230009740 0 0) 0x7fe0880abca0 con 0x7fe0880a98f0 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.345+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 >> v1:172.21.15.134:6790/0 conn(0x7fe0881430c0 legacy=0x7fe0881e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 >> v1:172.21.15.159:6789/0 conn(0x7fe08813f710 legacy=0x7fe088133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 1 monclient: found mon.a 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0881d8150 con 0x7fe0880a98f0 2024-03-25T19:36:15.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 10 monclient: _finish_auth 0 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe07effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 10 monclient: _renew_subs 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe088138eb0 con 0x7fe0880a98f0 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 10 monclient: _renew_subs 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe088137b20 con 0x7fe0880a98f0 2024-03-25T19:36:15.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.346+0000 7fe0906ed640 1 librados: init done 2024-03-25T19:36:15.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.347+0000 7fe0906ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:15.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.347+0000 7fe0906ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:15.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.347+0000 7fe0906ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:15.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.347+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0880627b0 con 0x7fe0880a98f0 2024-03-25T19:36:15.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.683+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe084004a90 con 0x7fe0880a98f0 2024-03-25T19:36:15.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.683+0000 7fe07effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:15.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.683+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fe084004da0 con 0x7fe0880a98f0 2024-03-25T19:36:15.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.683+0000 7fe07effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:15.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.683+0000 7fe07effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:15.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.683+0000 7fe07effd640 10 monclient: dump: 2024-03-25T19:36:15.683 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:15.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.683+0000 7fe08cc86640 4 set_mon_vals no callback set 2024-03-25T19:36:15.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.684+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fe08401ad70 con 0x7fe0880a98f0 2024-03-25T19:36:15.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.684+0000 7fe08dc88640 10 monclient: get_auth_request con 0x7fe06c06ddd0 auth_method 0 2024-03-25T19:36:15.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.685+0000 7fe07effd640 10 client.4602.objecter ms_handle_connect 0x7fe06c06ddd0 2024-03-25T19:36:15.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fe084054870 con 0x7fe0880a98f0 2024-03-25T19:36:15.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 10 client.4602.objecter ms_dispatch 0x7fe088000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:15.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 3 client.4602.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:15.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 3 client.4602.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:15.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-25T19:36:15.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe084050380 con 0x7fe0880a98f0 2024-03-25T19:36:15.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:15.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.690+0000 7fe07effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:15.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.945+0000 7fe0906ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:15.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.945+0000 7fe0906ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:15.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.945+0000 7fe0906ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:15.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.945+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0880b71a0 con 0x7fe0880a98f0 2024-03-25T19:36:15.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.973+0000 7fe07effd640 1 -- 172.21.15.134:0/1142092102 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fe0840522b0 con 0x7fe0880a98f0 2024-03-25T19:36:15.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.973+0000 7fe07effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:15.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.973+0000 7fe07effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:15.974 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:15.974 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:15.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 10 librados: watch_flush enter 2024-03-25T19:36:15.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 10 librados: watch_flush exit 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 10 client.4602.objecter successfully canceled tick 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 >> v1:172.21.15.159:6800/1875874887 conn(0x7fe06c06ddd0 legacy=0x7fe06c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 10 monclient: shutdown 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 >> v1:172.21.15.134:6789/0 conn(0x7fe0880a98f0 legacy=0x7fe0881d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 shutdown_connections 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 >> 172.21.15.134:0/1142092102 conn(0x7fe0880b7ea0 msgr2=0x7fe088074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 shutdown_connections 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 1 -- 172.21.15.134:0/1142092102 wait complete. 2024-03-25T19:36:15.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:15.976+0000 7fe0906ed640 1 librados: shutdown 2024-03-25T19:36:15.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:22.793 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:22.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.909+0000 7f83192fb640 10 monclient: get_monmap_and_config 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.909+0000 7f83192fb640 10 monclient: build_initial_monmap 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.909+0000 7f83192fb640 10 monclient: monmap: 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:36:22.910203+0000 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:36:22.910203+0000 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:36:22.909 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 1 Processor -- start 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 1 -- start start 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient: init 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient(hunting): picked mon.noname-a con 0x7f83140a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient(hunting): picked mon.noname-b con 0x7f831413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient(hunting): picked mon.noname-c con 0x7f83141430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8314142520 con 0x7f83140a98f0 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83140b28a0 con 0x7f83141430c0 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83140b2a30 con 0x7f831413f710 2024-03-25T19:36:22.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:22.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.910+0000 7f83192fb640 10 monclient(hunting): authenticate will time out at 6415.069571s 2024-03-25T19:36:22.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.918+0000 7f8312ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f83140a98f0 0x7f83140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:51622/0 (socket says 172.21.15.134:51622) 2024-03-25T19:36:22.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.918+0000 7f8312ffd640 1 -- 172.21.15.134:0/36321732 learned_addr learned my addr 172.21.15.134:0/36321732 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:22.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.928+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404430729 0 0) 0x7f83140b28a0 con 0x7f83141430c0 2024-03-25T19:36:22.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.928+0000 7f8311ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:22.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.928+0000 7f8311ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:22.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.928+0000 7f8311ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:22.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.928+0000 7f8311ffb640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:36:22.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.928+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82fc003520 con 0x7f83141430c0 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.938+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153650172 0 0) 0x7f82fc003520 con 0x7f83141430c0 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.938+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 >> v1:172.21.15.134:6790/0 conn(0x7f831413f710 legacy=0x7f83141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.938+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 >> v1:172.21.15.134:6789/0 conn(0x7f83140a98f0 legacy=0x7f83140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 1 monclient: found mon.noname-c 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83140b44c0 con 0x7f83141430c0 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f830c001a10 con 0x7f83141430c0 2024-03-25T19:36:22.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f83192fb640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f83192fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f8311ffb640 10 monclient: dump: 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:22.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.939+0000 7f83192fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:22.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f830c001c30 con 0x7f83141430c0 2024-03-25T19:36:22.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f8311ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:22.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f8311ffb640 1 -- 172.21.15.134:0/36321732 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f830c001fc0 con 0x7f83141430c0 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f83192fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f83192fb640 4 set_mon_vals no callback set 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f8311ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f8311ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.947+0000 7f8311ffb640 10 monclient: dump: 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:22.948 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.948+0000 7f83192fb640 10 monclient: shutdown 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.948+0000 7f83192fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.948+0000 7f83192fb640 1 -- 172.21.15.134:0/36321732 >> v1:172.21.15.159:6789/0 conn(0x7f83141430c0 legacy=0x7f831414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.948+0000 7f83192fb640 1 -- 172.21.15.134:0/36321732 shutdown_connections 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.948+0000 7f83192fb640 1 -- 172.21.15.134:0/36321732 >> 172.21.15.134:0/36321732 conn(0x7f83140b7ea0 msgr2=0x7f83140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.948+0000 7f83192fb640 1 -- 172.21.15.134:0/36321732 shutdown_connections 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.948+0000 7f83192fb640 1 -- 172.21.15.134:0/36321732 wait complete. 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.949+0000 7f83192fb640 10 monclient: build_initial_monmap 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.949+0000 7f83192fb640 10 monclient: monmap: 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:22.950 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.949+0000 7f83192fb640 1 librados: starting msgr at 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 1 librados: starting objecter 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 1 Processor -- start 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 1 -- start start 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 1 librados: setting wanted keys 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 1 librados: calling monclient init 2024-03-25T19:36:22.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 10 monclient: init 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.950+0000 7f83192fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 10 monclient(hunting): picked mon.noname-a con 0x7f83140a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 10 monclient(hunting): picked mon.noname-b con 0x7f831413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 10 monclient(hunting): picked mon.noname-c con 0x7f83141430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83141396e0 con 0x7f83140a98f0 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8314138530 con 0x7f831413f710 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83141399c0 con 0x7f83141430c0 2024-03-25T19:36:22.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:22.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.951+0000 7f83192fb640 10 monclient(hunting): authenticate will time out at 6415.109690s 2024-03-25T19:36:22.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.966+0000 7f83137fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f83141430c0 0x7f83141e1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:42814/0 (socket says 172.21.15.134:42814) 2024-03-25T19:36:22.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.966+0000 7f83137fe640 1 -- 172.21.15.134:0/3758377086 learned_addr learned my addr 172.21.15.134:0/3758377086 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:22.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.969+0000 7f82fb7fe640 10 client.?.objecter ms_handle_connect 0x7f83141430c0 2024-03-25T19:36:22.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.969+0000 7f82fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:22.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.971+0000 7f82fb7fe640 10 client.?.objecter ms_handle_connect 0x7f831413f710 2024-03-25T19:36:22.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.971+0000 7f82fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:22.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.976+0000 7f82fb7fe640 10 client.?.objecter ms_handle_connect 0x7f83140a98f0 2024-03-25T19:36:22.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.976+0000 7f82fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:22.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251237460 0 0) 0x7f8314138530 con 0x7f831413f710 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8300004eb0 con 0x7f831413f710 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341117767 0 0) 0x7f83141399c0 con 0x7f83141430c0 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:22.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:36:22.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.978+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8314138530 con 0x7f83141430c0 2024-03-25T19:36:22.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.987+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028997818 0 0) 0x7f8300004eb0 con 0x7f831413f710 2024-03-25T19:36:22.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.987+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83141399c0 con 0x7f831413f710 2024-03-25T19:36:22.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.987+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457793987 0 0) 0x7f83141396e0 con 0x7f83140a98f0 2024-03-25T19:36:22.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.987+0000 7f82fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:22.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.987+0000 7f82fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:22.988 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.988+0000 7f82fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:22.988 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.988+0000 7f82fb7fe640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:36:22.988 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.988+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8300004eb0 con 0x7f83140a98f0 2024-03-25T19:36:22.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.991+0000 7f83127fc640 1 -- 172.21.15.134:0/3758377086 >> v1:172.21.15.159:6789/0 conn(0x7f831413f710 legacy=0x7f83141da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T19:36:22.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.991+0000 7f83127fc640 1 -- 172.21.15.134:0/3758377086 >> v1:172.21.15.159:6789/0 conn(0x7f831413f710 legacy=0x7f83141da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:36:22.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.991+0000 7f83127fc640 1 --1- 172.21.15.134:0/3758377086 >> v1:172.21.15.159:6789/0 conn(0x7f831413f710 0x7f83141da850 :-1 s=OPENED pgs=58589 cs=1 l=1).handle_message read tag failed 2024-03-25T19:36:22.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.991+0000 7f83127fc640 1 --1- 172.21.15.134:0/3758377086 >> v1:172.21.15.159:6789/0 conn(0x7f831413f710 0x7f83141da850 :-1 s=OPENED pgs=58589 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:36:22.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.991+0000 7f82fb7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.159:6789/0 2024-03-25T19:36:22.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.996+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4197519027 0 0) 0x7f8300004eb0 con 0x7f83140a98f0 2024-03-25T19:36:22.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:22.996+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83141396e0 con 0x7f83140a98f0 2024-03-25T19:36:23.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.000+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8304001780 con 0x7f83140a98f0 2024-03-25T19:36:23.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.001+0000 7f82fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.001+0000 7f82fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.001+0000 7f82fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:23.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.001+0000 7f82fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.006+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1382066221 0 0) 0x7f8314138530 con 0x7f83141430c0 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.006+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8300004eb0 con 0x7f83141430c0 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.006+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f830c001f40 con 0x7f83141430c0 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.006+0000 7f82fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.006+0000 7f82fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.006+0000 7f82fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:23.006 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.007 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:23.007 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:23.007 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:23.007 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:23.007 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:23.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1085519042 0 0) 0x7f83141396e0 con 0x7f83140a98f0 2024-03-25T19:36:23.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 >> v1:172.21.15.134:6790/0 conn(0x7f83141430c0 legacy=0x7f83141e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 >> v1:172.21.15.159:6789/0 conn(0x7f831413f710 legacy=0x7f83141da850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 1 monclient: found mon.a 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8314138a50 con 0x7f83140a98f0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f82fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 5 monclient: authenticate success, global_id 4608 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 10 monclient: _renew_subs 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83141380a0 con 0x7f83140a98f0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 10 monclient: _renew_subs 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83141db140 con 0x7f83140a98f0 2024-03-25T19:36:23.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.012+0000 7f83192fb640 1 librados: init done 2024-03-25T19:36:23.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.013+0000 7f83192fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:23.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.013+0000 7f83192fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:23.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.013+0000 7f83192fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:23.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.013+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8314138ca0 con 0x7f83140a98f0 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.321+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8304001c50 con 0x7f83140a98f0 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.321+0000 7f82fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.321+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f83040047e0 con 0x7f83140a98f0 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.321+0000 7f82fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.321+0000 7f82fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.321+0000 7f82fb7fe640 10 monclient: dump: 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:23.322 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:23.323 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:23.323 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:23.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.321+0000 7f83117fa640 4 set_mon_vals no callback set 2024-03-25T19:36:23.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.322+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f8304004a00 con 0x7f83140a98f0 2024-03-25T19:36:23.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.323+0000 7f83127fc640 10 monclient: get_auth_request con 0x7f830006e1a0 auth_method 0 2024-03-25T19:36:23.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.323+0000 7f82fb7fe640 10 client.4608.objecter ms_handle_connect 0x7f830006e1a0 2024-03-25T19:36:23.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f83040544f0 con 0x7f83140a98f0 2024-03-25T19:36:23.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 10 client.4608.objecter ms_dispatch 0x7f8314000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:23.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 3 client.4608.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:23.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 3 client.4608.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:23.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-25T19:36:23.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8304050000 con 0x7f83140a98f0 2024-03-25T19:36:23.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:23.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.328+0000 7f82fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:23.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.551+0000 7f83192fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:23.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.551+0000 7f83192fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:23.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.551+0000 7f83192fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:36:23.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.551+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8314138720 con 0x7f83140a98f0 2024-03-25T19:36:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.577+0000 7f82fb7fe640 1 -- 172.21.15.134:0/3758377086 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f8304051f30 con 0x7f83140a98f0 2024-03-25T19:36:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.577+0000 7f82fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:23.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.577+0000 7f82fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:23.578 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:23.578 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 10 librados: watch_flush enter 2024-03-25T19:36:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 10 librados: watch_flush exit 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 10 client.4608.objecter successfully canceled tick 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 >> v1:172.21.15.159:6800/1875874887 conn(0x7f830006e1a0 legacy=0x7f8300070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 10 monclient: shutdown 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 >> v1:172.21.15.134:6789/0 conn(0x7f83140a98f0 legacy=0x7f8314137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 shutdown_connections 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 >> 172.21.15.134:0/3758377086 conn(0x7f83140b7ea0 msgr2=0x7f8314074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 shutdown_connections 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 1 -- 172.21.15.134:0/3758377086 wait complete. 2024-03-25T19:36:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:23.580+0000 7f83192fb640 1 librados: shutdown 2024-03-25T19:36:23.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:26.189 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:30.861 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.199 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:30.872 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.235 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:30.907 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.243 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:30.915 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.250 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:30.922 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.259 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:30.906 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.266 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:30.913 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.286 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:30.959 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.298 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:30.945 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:30.399 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:30.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.511+0000 7fd3b38c9640 10 monclient: get_monmap_and_config 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.511+0000 7fd3b38c9640 10 monclient: build_initial_monmap 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.511+0000 7fd3b38c9640 10 monclient: monmap: 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:36:30.512167+0000 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:36:30.512167+0000 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:36:30.511 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.511+0000 7fd3b38c9640 1 Processor -- start 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.511+0000 7fd3b38c9640 1 -- start start 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.511+0000 7fd3b38c9640 10 monclient: init 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ac0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ac14be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0af940 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0b0a70 con 0x7fd3ac14be90 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0b0d50 con 0x7fd3ac0b1730 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:30.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.512+0000 7fd3b38c9640 10 monclient(hunting): authenticate will time out at 6422.671304s 2024-03-25T19:36:30.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.524+0000 7fd3b1665640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd3ac0b4d60 0x7fd3ac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:56258/0 (socket says 172.21.15.134:56258) 2024-03-25T19:36:30.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.524+0000 7fd3b1665640 1 -- 172.21.15.134:0/526545452 learned_addr learned my addr 172.21.15.134:0/526545452 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:30.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.541+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255859730 0 0) 0x7fd3ac0b0a70 con 0x7fd3ac14be90 2024-03-25T19:36:30.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.541+0000 7fd3a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:30.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.541+0000 7fd3a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:30.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.541+0000 7fd3a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:30.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.541+0000 7fd3a3fff640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:36:30.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.541+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd398003520 con 0x7fd3ac14be90 2024-03-25T19:36:30.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.552+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582324629 0 0) 0x7fd3ac0af940 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.552+0000 7fd3a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:30.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.552+0000 7fd3a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:30.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.552+0000 7fd3a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:30.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.552+0000 7fd3a3fff640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:36:30.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.552+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac0b0a70 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.555+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964402897 0 0) 0x7fd3ac0b0d50 con 0x7fd3ac0b1730 2024-03-25T19:36:30.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.555+0000 7fd3a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:30.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.555+0000 7fd3a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:30.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.555+0000 7fd3a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:30.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.555+0000 7fd3a3fff640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:36:30.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.555+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac0af940 con 0x7fd3ac0b1730 2024-03-25T19:36:30.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1398488738 0 0) 0x7fd398003520 con 0x7fd3ac14be90 2024-03-25T19:36:30.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 >> v1:172.21.15.134:6790/0 conn(0x7fd3ac0b1730 legacy=0x7fd3ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:30.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 >> v1:172.21.15.134:6789/0 conn(0x7fd3ac0b4d60 legacy=0x7fd3ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:30.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:30.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 1 monclient: found mon.noname-c 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0b1030 con 0x7fd3ac14be90 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd3a4001a10 con 0x7fd3ac14be90 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3a3fff640 10 monclient: dump: 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.565 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:30.566 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:30.566 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:30.566 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:30.566 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:30.566 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3b38c9640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:36:30.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.564+0000 7fd3b38c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:30.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3a4001b70 con 0x7fd3ac14be90 2024-03-25T19:36:30.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3a3fff640 1 -- 172.21.15.134:0/526545452 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd3a4001f80 con 0x7fd3ac14be90 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3a3fff640 10 monclient: dump: 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:30.591 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3b38c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:30.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.591+0000 7fd3b38c9640 4 set_mon_vals no callback set 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 10 monclient: shutdown 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 1 -- 172.21.15.134:0/526545452 >> v1:172.21.15.159:6789/0 conn(0x7fd3ac14be90 legacy=0x7fd3ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 1 -- 172.21.15.134:0/526545452 shutdown_connections 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 1 -- 172.21.15.134:0/526545452 >> 172.21.15.134:0/526545452 conn(0x7fd3ac0617a0 msgr2=0x7fd3ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 1 -- 172.21.15.134:0/526545452 shutdown_connections 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 1 -- 172.21.15.134:0/526545452 wait complete. 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 10 monclient: build_initial_monmap 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.592+0000 7fd3b38c9640 10 monclient: monmap: 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:36:30.593 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.593+0000 7fd3b38c9640 1 librados: starting msgr at 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.593+0000 7fd3b38c9640 1 librados: starting objecter 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.593+0000 7fd3b38c9640 1 Processor -- start 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.593+0000 7fd3b38c9640 1 -- start start 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.593+0000 7fd3b38c9640 1 librados: setting wanted keys 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.593+0000 7fd3b38c9640 1 librados: calling monclient init 2024-03-25T19:36:30.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.593+0000 7fd3b38c9640 10 monclient: init 2024-03-25T19:36:30.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:30.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:30.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:30.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ac0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:30.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ac14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:30.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac135ee0 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac135b00 con 0x7fd3ac0b1730 2024-03-25T19:36:30.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac135c90 con 0x7fd3ac14be90 2024-03-25T19:36:30.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:30.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.594+0000 7fd3b38c9640 10 monclient(hunting): authenticate will time out at 6422.753085s 2024-03-25T19:36:30.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.603+0000 7fd3b0e64640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd3ac0b4d60 0x7fd3ac1d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:56312/0 (socket says 172.21.15.134:56312) 2024-03-25T19:36:30.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.603+0000 7fd3b0e64640 1 -- 172.21.15.134:0/757495185 learned_addr learned my addr 172.21.15.134:0/757495185 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:30.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.607+0000 7fd3a1ffb640 10 client.?.objecter ms_handle_connect 0x7fd3ac0b4d60 2024-03-25T19:36:30.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.607+0000 7fd3a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:30.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.615+0000 7fd3a1ffb640 10 client.?.objecter ms_handle_connect 0x7fd3ac0b1730 2024-03-25T19:36:30.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.615+0000 7fd3a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:30.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.617+0000 7fd3a1ffb640 10 client.?.objecter ms_handle_connect 0x7fd3ac14be90 2024-03-25T19:36:30.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.617+0000 7fd3a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:30.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.625+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231964338 0 0) 0x7fd3ac135ee0 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.625+0000 7fd3a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:30.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.625+0000 7fd3a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:30.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.625+0000 7fd3a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:30.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.625+0000 7fd3a1ffb640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:36:30.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.625+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd390004eb0 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.634+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613625890 0 0) 0x7fd3ac135c90 con 0x7fd3ac14be90 2024-03-25T19:36:30.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.634+0000 7fd3a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:30.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.634+0000 7fd3a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:30.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.634+0000 7fd3a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:30.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.634+0000 7fd3a1ffb640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:36:30.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.634+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac135ee0 con 0x7fd3ac14be90 2024-03-25T19:36:30.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.636+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129268003 0 0) 0x7fd3ac135b00 con 0x7fd3ac0b1730 2024-03-25T19:36:30.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.636+0000 7fd3a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:30.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.636+0000 7fd3a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:30.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.636+0000 7fd3a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:30.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.636+0000 7fd3a1ffb640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:36:30.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.636+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac135c90 con 0x7fd3ac0b1730 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.648+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2186597361 0 0) 0x7fd390004eb0 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.648+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3ac135b00 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.648+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd39c001a10 con 0x7fd3ac0b4d60 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.648+0000 7fd3a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.648+0000 7fd3a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.648+0000 7fd3a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:30.648 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.648+0000 7fd3a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.649+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2908799747 0 0) 0x7fd3ac135ee0 con 0x7fd3ac14be90 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.649+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd390004eb0 con 0x7fd3ac14be90 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.649+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd3a4001e80 con 0x7fd3ac14be90 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.649+0000 7fd3a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.649+0000 7fd3a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:30.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.649+0000 7fd3a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:30.650 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.662+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 615502313 0 0) 0x7fd3ac135c90 con 0x7fd3ac0b1730 2024-03-25T19:36:30.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.662+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3ac135ee0 con 0x7fd3ac0b1730 2024-03-25T19:36:30.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.662+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd3a8001930 con 0x7fd3ac0b1730 2024-03-25T19:36:30.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.662+0000 7fd3a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:30.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.662+0000 7fd3a1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:30.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.662+0000 7fd3a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:30.662 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:30.663 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 450680108 0 0) 0x7fd390004eb0 con 0x7fd3ac14be90 2024-03-25T19:36:30.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 >> v1:172.21.15.159:6789/0 conn(0x7fd3ac0b1730 legacy=0x7fd3ac12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 >> v1:172.21.15.134:6789/0 conn(0x7fd3ac0b4d60 legacy=0x7fd3ac1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 1 monclient: found mon.c 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac136960 con 0x7fd3ac14be90 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 5 monclient: authenticate success, global_id 4577 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 10 monclient: _renew_subs 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac130470 con 0x7fd3ac14be90 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 10 monclient: _renew_subs 2024-03-25T19:36:30.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:30.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac13e690 con 0x7fd3ac14be90 2024-03-25T19:36:30.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.665+0000 7fd3b38c9640 1 librados: init done 2024-03-25T19:36:30.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.666+0000 7fd3b38c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:30.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.666+0000 7fd3b38c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:30.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.666+0000 7fd3b38c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:30.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.666+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac136b50 con 0x7fd3ac14be90 2024-03-25T19:36:30.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.959+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3a4001fe0 con 0x7fd3ac14be90 2024-03-25T19:36:30.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.959+0000 7fd3a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.959+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fd3a4003c20 con 0x7fd3ac14be90 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.959+0000 7fd3a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.959+0000 7fd3a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.959+0000 7fd3a1ffb640 10 monclient: dump: 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:30.960 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:30.961 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:30.961 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:30.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.959+0000 7fd3a37fe640 4 set_mon_vals no callback set 2024-03-25T19:36:30.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.960+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fd3a4019bf0 con 0x7fd3ac14be90 2024-03-25T19:36:30.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.961+0000 7fd3b1665640 10 monclient: get_auth_request con 0x7fd39006e110 auth_method 0 2024-03-25T19:36:30.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.961+0000 7fd3a1ffb640 10 client.4577.objecter ms_handle_connect 0x7fd39006e110 2024-03-25T19:36:30.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.965+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fd3a40013e0 con 0x7fd3ac14be90 2024-03-25T19:36:30.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.965+0000 7fd3a1ffb640 10 client.4577.objecter ms_dispatch 0x7fd3ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:30.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.965+0000 7fd3a1ffb640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:30.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.965+0000 7fd3a1ffb640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:30.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.965+0000 7fd3a1ffb640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-25T19:36:30.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.965+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3a404efd0 con 0x7fd3ac14be90 2024-03-25T19:36:30.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.966+0000 7fd3a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:30.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:30.966+0000 7fd3a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:31.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.212+0000 7fd3b38c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:31.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.212+0000 7fd3b38c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:31.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.212+0000 7fd3b38c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:31.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.212+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3ac0627b0 con 0x7fd3ac14be90 2024-03-25T19:36:31.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.242+0000 7fd3a1ffb640 1 -- 172.21.15.134:0/757495185 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fd3a40510d0 con 0x7fd3ac14be90 2024-03-25T19:36:31.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.242+0000 7fd3a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:31.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.242+0000 7fd3a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:31.243 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:31.243 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:31.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 10 librados: watch_flush enter 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 10 librados: watch_flush exit 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 10 client.4577.objecter successfully canceled tick 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 >> v1:172.21.15.159:6800/1875874887 conn(0x7fd39006e110 legacy=0x7fd3900705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 10 monclient: shutdown 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 >> v1:172.21.15.134:6790/0 conn(0x7fd3ac14be90 legacy=0x7fd3ac1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 shutdown_connections 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 >> 172.21.15.134:0/757495185 conn(0x7fd3ac0617a0 msgr2=0x7fd3ac0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 shutdown_connections 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 1 -- 172.21.15.134:0/757495185 wait complete. 2024-03-25T19:36:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:31.245+0000 7fd3b38c9640 1 librados: shutdown 2024-03-25T19:36:31.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:38.063 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:38.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.181+0000 7f9a149a8640 10 monclient: get_monmap_and_config 2024-03-25T19:36:38.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.181+0000 7f9a149a8640 10 monclient: build_initial_monmap 2024-03-25T19:36:38.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.181+0000 7f9a149a8640 10 monclient: monmap: 2024-03-25T19:36:38.181 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:38.181 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:36:38.182848+0000 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:36:38.182848+0000 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:36:38.182 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:38.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.182+0000 7f9a149a8640 1 Processor -- start 2024-03-25T19:36:38.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.182+0000 7f9a149a8640 1 -- start start 2024-03-25T19:36:38.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.182+0000 7f9a149a8640 10 monclient: init 2024-03-25T19:36:38.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:38.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9a100a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9a1013f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9a101430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10142520 con 0x7f9a100a98f0 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a100b28a0 con 0x7f9a1013f710 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a100b2a30 con 0x7f9a101430c0 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:38.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.183+0000 7f9a149a8640 10 monclient(hunting): authenticate will time out at 6430.341832s 2024-03-25T19:36:38.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.186+0000 7f9a05d9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9a1013f710 0x7f9a101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:37148/0 (socket says 172.21.15.134:37148) 2024-03-25T19:36:38.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.186+0000 7f9a05d9b640 1 -- 172.21.15.134:0/2625419867 learned_addr learned my addr 172.21.15.134:0/2625419867 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:38.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.195+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764843963 0 0) 0x7f9a100b28a0 con 0x7f9a1013f710 2024-03-25T19:36:38.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.195+0000 7f9a0dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:38.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.195+0000 7f9a0dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:38.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.195+0000 7f9a0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:38.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.195+0000 7f9a0dd9b640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:36:38.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.195+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99fc003520 con 0x7f9a1013f710 2024-03-25T19:36:38.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.206+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494532243 0 0) 0x7f9a10142520 con 0x7f9a100a98f0 2024-03-25T19:36:38.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.206+0000 7f9a0dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:38.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.206+0000 7f9a0dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:38.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.206+0000 7f9a0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:38.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.206+0000 7f9a0dd9b640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:36:38.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.206+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a100b28a0 con 0x7f9a100a98f0 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939979170 0 0) 0x7f99fc003520 con 0x7f9a1013f710 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 >> v1:172.21.15.134:6790/0 conn(0x7f9a101430c0 legacy=0x7f9a1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 >> v1:172.21.15.134:6789/0 conn(0x7f9a100a98f0 legacy=0x7f9a100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a100b44c0 con 0x7f9a1013f710 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f99f8001690 con 0x7f9a1013f710 2024-03-25T19:36:38.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a0dd9b640 10 monclient: dump: 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a149a8640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:36:38.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.210+0000 7f9a149a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:38.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99f80017f0 con 0x7f9a1013f710 2024-03-25T19:36:38.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a0dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:38.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a0dd9b640 1 -- 172.21.15.134:0/2625419867 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f99f8001c00 con 0x7f9a1013f710 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a0dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a0dd9b640 10 monclient: dump: 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a149a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:38.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.212+0000 7f9a149a8640 4 set_mon_vals no callback set 2024-03-25T19:36:38.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.213+0000 7f9a149a8640 10 monclient: shutdown 2024-03-25T19:36:38.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.213+0000 7f9a149a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.213+0000 7f9a149a8640 1 -- 172.21.15.134:0/2625419867 >> v1:172.21.15.159:6789/0 conn(0x7f9a1013f710 legacy=0x7f9a101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.213+0000 7f9a149a8640 1 -- 172.21.15.134:0/2625419867 shutdown_connections 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.213+0000 7f9a149a8640 1 -- 172.21.15.134:0/2625419867 >> 172.21.15.134:0/2625419867 conn(0x7f9a100b7ea0 msgr2=0x7f9a100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.213+0000 7f9a149a8640 1 -- 172.21.15.134:0/2625419867 shutdown_connections 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.213+0000 7f9a149a8640 1 -- 172.21.15.134:0/2625419867 wait complete. 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.214+0000 7f9a149a8640 10 monclient: build_initial_monmap 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.214+0000 7f9a149a8640 10 monclient: monmap: 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:38.215 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:36:38.216 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:36:38.216 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:38.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 1 librados: starting msgr at 2024-03-25T19:36:38.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 1 librados: starting objecter 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 1 Processor -- start 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 1 -- start start 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 1 librados: setting wanted keys 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 1 librados: calling monclient init 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 10 monclient: init 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9a100a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.215+0000 7f9a149a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9a1013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.216+0000 7f9a149a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9a101430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.216+0000 7f9a149a8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a100abdc0 con 0x7f9a1013f710 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.216+0000 7f9a149a8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a101d7730 con 0x7f9a100a98f0 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.216+0000 7f9a149a8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10138f10 con 0x7f9a101430c0 2024-03-25T19:36:38.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.216+0000 7f9a149a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:38.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.216+0000 7f9a149a8640 10 monclient(hunting): authenticate will time out at 6430.374672s 2024-03-25T19:36:38.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.223+0000 7f9a0e59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9a100a98f0 0x7f9a10138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:37178/0 (socket says 172.21.15.134:37178) 2024-03-25T19:36:38.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.223+0000 7f9a0e59c640 1 -- 172.21.15.134:0/4274164681 learned_addr learned my addr 172.21.15.134:0/4274164681 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:38.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.224+0000 7f9a077fe640 10 client.?.objecter ms_handle_connect 0x7f9a100a98f0 2024-03-25T19:36:38.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.224+0000 7f9a077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:38.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.231+0000 7f9a077fe640 10 client.?.objecter ms_handle_connect 0x7f9a101430c0 2024-03-25T19:36:38.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.231+0000 7f9a077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:38.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 10 client.?.objecter ms_handle_connect 0x7f9a1013f710 2024-03-25T19:36:38.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:38.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262317446 0 0) 0x7f9a101d7730 con 0x7f9a100a98f0 2024-03-25T19:36:38.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:38.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:38.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:38.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:36:38.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.241+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f0004eb0 con 0x7f9a100a98f0 2024-03-25T19:36:38.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.244+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795698449 0 0) 0x7f9a10138f10 con 0x7f9a101430c0 2024-03-25T19:36:38.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.244+0000 7f9a077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:38.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.244+0000 7f9a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:38.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.244+0000 7f9a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:38.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.244+0000 7f9a077fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:36:38.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.244+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a101d7730 con 0x7f9a101430c0 2024-03-25T19:36:38.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.249+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3714842888 0 0) 0x7f99f0004eb0 con 0x7f9a100a98f0 2024-03-25T19:36:38.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.249+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a10138f10 con 0x7f9a100a98f0 2024-03-25T19:36:38.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.249+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9a00001a10 con 0x7f9a100a98f0 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.249+0000 7f9a077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.249+0000 7f9a077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.249+0000 7f9a077fe640 10 monclient(hunting): dump: 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:38.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.249+0000 7f9a077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:38.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.267+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606546275 0 0) 0x7f9a100abdc0 con 0x7f9a1013f710 2024-03-25T19:36:38.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.267+0000 7f9a077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:38.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.267+0000 7f9a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:38.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.267+0000 7f9a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:38.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.267+0000 7f9a077fe640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:36:38.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.267+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f0004eb0 con 0x7f9a1013f710 2024-03-25T19:36:38.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3543999508 0 0) 0x7f9a10138f10 con 0x7f9a100a98f0 2024-03-25T19:36:38.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 >> v1:172.21.15.134:6790/0 conn(0x7f9a101430c0 legacy=0x7f9a101d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:38.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 >> v1:172.21.15.134:6789/0 conn(0x7f9a1013f710 legacy=0x7f9a101d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:38.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 1 monclient: found mon.b 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a101390a0 con 0x7f9a100a98f0 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 5 monclient: authenticate success, global_id 4576 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 10 monclient: _renew_subs 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a101d8320 con 0x7f9a100a98f0 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 10 monclient: _renew_subs 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:36:38.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a101d43c0 con 0x7f9a100a98f0 2024-03-25T19:36:38.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.272+0000 7f9a149a8640 1 librados: init done 2024-03-25T19:36:38.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.273+0000 7f9a149a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:38.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.273+0000 7f9a149a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:38.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.273+0000 7f9a149a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:36:38.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.273+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a100627b0 con 0x7f9a100a98f0 2024-03-25T19:36:38.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.551+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a00001c50 con 0x7f9a100a98f0 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.551+0000 7f9a077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.551+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9a00003a60 con 0x7f9a100a98f0 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.551+0000 7f9a077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.551+0000 7f9a077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.551+0000 7f9a077fe640 10 monclient: dump: 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:38.551 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:38.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.551+0000 7f9a0d59a640 4 set_mon_vals no callback set 2024-03-25T19:36:38.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.552+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f9a00003c80 con 0x7f9a100a98f0 2024-03-25T19:36:38.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.552+0000 7f9a05d9b640 10 monclient: get_auth_request con 0x7f99f006de00 auth_method 0 2024-03-25T19:36:38.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.553+0000 7f9a077fe640 10 client.4576.objecter ms_handle_connect 0x7f99f006de00 2024-03-25T19:36:38.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.557+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f9a00054860 con 0x7f9a100a98f0 2024-03-25T19:36:38.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.557+0000 7f9a077fe640 10 client.4576.objecter ms_dispatch 0x7f9a10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:38.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.557+0000 7f9a077fe640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:38.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.557+0000 7f9a077fe640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:38.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.557+0000 7f9a077fe640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T19:36:38.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.559+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a00050370 con 0x7f9a100a98f0 2024-03-25T19:36:38.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.559+0000 7f9a077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:38.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.559+0000 7f9a077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:38.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.777+0000 7f9a149a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:38.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.777+0000 7f9a149a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:38.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.777+0000 7f9a149a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:36:38.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.777+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a10069f40 con 0x7f9a100a98f0 2024-03-25T19:36:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.845+0000 7f9a077fe640 1 -- 172.21.15.134:0/4274164681 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f9a000522a0 con 0x7f9a100a98f0 2024-03-25T19:36:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.845+0000 7f9a077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.845+0000 7f9a077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:38.845 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:38.845 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 10 librados: watch_flush enter 2024-03-25T19:36:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 10 librados: watch_flush exit 2024-03-25T19:36:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T19:36:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 10 client.4576.objecter successfully canceled tick 2024-03-25T19:36:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 >> v1:172.21.15.159:6800/1875874887 conn(0x7f99f006de00 legacy=0x7f99f00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 10 monclient: shutdown 2024-03-25T19:36:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 >> v1:172.21.15.159:6789/0 conn(0x7f9a100a98f0 legacy=0x7f9a10138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 shutdown_connections 2024-03-25T19:36:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 >> 172.21.15.134:0/4274164681 conn(0x7f9a100b7ea0 msgr2=0x7f9a10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 shutdown_connections 2024-03-25T19:36:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 1 -- 172.21.15.134:0/4274164681 wait complete. 2024-03-25T19:36:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:38.847+0000 7f9a149a8640 1 librados: shutdown 2024-03-25T19:36:38.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:45.665 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:45.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.780+0000 7fee9aab3640 10 monclient: get_monmap_and_config 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.780+0000 7fee9aab3640 10 monclient: build_initial_monmap 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.780+0000 7fee9aab3640 10 monclient: monmap: 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:36:45.781167+0000 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:36:45.781167+0000 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:36:45.780 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:45.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.780+0000 7fee9aab3640 1 Processor -- start 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.780+0000 7fee9aab3640 1 -- start start 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.780+0000 7fee9aab3640 10 monclient: init 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 10 monclient(hunting): picked mon.noname-a con 0x7fee940a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 10 monclient(hunting): picked mon.noname-c con 0x7fee9413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 10 monclient(hunting): picked mon.noname-b con 0x7fee941430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee94142520 con 0x7fee940a98f0 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee940b28a0 con 0x7fee9413f710 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee940b2a30 con 0x7fee941430c0 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:45.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.781+0000 7fee9aab3640 10 monclient(hunting): authenticate will time out at 6437.940257s 2024-03-25T19:36:45.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.791+0000 7fee99050640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fee941430c0 0x7fee9414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:48328/0 (socket says 172.21.15.134:48328) 2024-03-25T19:36:45.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.791+0000 7fee99050640 1 -- 172.21.15.134:0/229639086 learned_addr learned my addr 172.21.15.134:0/229639086 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:45.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.798+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769034634 0 0) 0x7fee940b2a30 con 0x7fee941430c0 2024-03-25T19:36:45.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.798+0000 7fee937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:45.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.798+0000 7fee937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:45.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.798+0000 7fee937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:45.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.798+0000 7fee937fe640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:36:45.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.798+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee84003520 con 0x7fee941430c0 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299811207 0 0) 0x7fee84003520 con 0x7fee941430c0 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 >> v1:172.21.15.159:6789/0 conn(0x7fee9413f710 legacy=0x7fee941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 >> v1:172.21.15.134:6789/0 conn(0x7fee940a98f0 legacy=0x7fee940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 1 monclient: found mon.noname-b 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee940b44c0 con 0x7fee941430c0 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:45.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee8c001a10 con 0x7fee941430c0 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee9aab3640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee9aab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee937fe640 10 monclient: dump: 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:45.815 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:45.816 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:45.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.814+0000 7fee9aab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:45.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee8c001b70 con 0x7fee941430c0 2024-03-25T19:36:45.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:45.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee937fe640 1 -- 172.21.15.134:0/229639086 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee8c001f80 con 0x7fee941430c0 2024-03-25T19:36:45.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee9aab3640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:45.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee9aab3640 4 set_mon_vals no callback set 2024-03-25T19:36:45.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:45.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.826+0000 7fee937fe640 10 monclient: dump: 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:45.827 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:45.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.827+0000 7fee9aab3640 10 monclient: shutdown 2024-03-25T19:36:45.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.827+0000 7fee9aab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:45.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.827+0000 7fee9aab3640 1 -- 172.21.15.134:0/229639086 >> v1:172.21.15.134:6790/0 conn(0x7fee941430c0 legacy=0x7fee9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:45.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.827+0000 7fee9aab3640 1 -- 172.21.15.134:0/229639086 shutdown_connections 2024-03-25T19:36:45.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.827+0000 7fee9aab3640 1 -- 172.21.15.134:0/229639086 >> 172.21.15.134:0/229639086 conn(0x7fee940b7ea0 msgr2=0x7fee940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:45.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.827+0000 7fee9aab3640 1 -- 172.21.15.134:0/229639086 shutdown_connections 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.827+0000 7fee9aab3640 1 -- 172.21.15.134:0/229639086 wait complete. 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.828+0000 7fee9aab3640 10 monclient: build_initial_monmap 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.828+0000 7fee9aab3640 10 monclient: monmap: 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.828+0000 7fee9aab3640 1 librados: starting msgr at 2024-03-25T19:36:45.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.828+0000 7fee9aab3640 1 librados: starting objecter 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.829+0000 7fee9aab3640 1 Processor -- start 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.829+0000 7fee9aab3640 1 -- start start 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.829+0000 7fee9aab3640 1 librados: setting wanted keys 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.829+0000 7fee9aab3640 1 librados: calling monclient init 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.829+0000 7fee9aab3640 10 monclient: init 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 10 monclient(hunting): picked mon.noname-c con 0x7fee940a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 10 monclient(hunting): picked mon.noname-b con 0x7fee9413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 10 monclient(hunting): picked mon.noname-a con 0x7fee941430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee941d7500 con 0x7fee941430c0 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee941394a0 con 0x7fee9413f710 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee94138b50 con 0x7fee940a98f0 2024-03-25T19:36:45.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:45.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.830+0000 7fee9aab3640 10 monclient(hunting): authenticate will time out at 6437.989233s 2024-03-25T19:36:45.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.846+0000 7fee9884f640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fee940a98f0 0x7fee94138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:48376/0 (socket says 172.21.15.134:48376) 2024-03-25T19:36:45.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.846+0000 7fee9884f640 1 -- 172.21.15.134:0/1691307299 learned_addr learned my addr 172.21.15.134:0/1691307299 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:45.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.848+0000 7fee99050640 1 -- 172.21.15.134:0/1691307299 >> v1:172.21.15.134:6789/0 conn(0x7fee941430c0 legacy=0x7fee94134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:36:45.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.848+0000 7fee99050640 1 -- 172.21.15.134:0/1691307299 >> v1:172.21.15.134:6789/0 conn(0x7fee941430c0 legacy=0x7fee94134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:36:45.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.848+0000 7fee99050640 1 --1- 172.21.15.134:0/1691307299 >> v1:172.21.15.134:6789/0 conn(0x7fee941430c0 0x7fee94134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T19:36:45.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.848+0000 7fee99050640 1 --1- 172.21.15.134:0/1691307299 >> v1:172.21.15.134:6789/0 conn(0x7fee941430c0 0x7fee94134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:36:45.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.848+0000 7fee917fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.134:6789/0 2024-03-25T19:36:45.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.850+0000 7fee917fa640 10 client.?.objecter ms_handle_connect 0x7fee940a98f0 2024-03-25T19:36:45.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.850+0000 7fee917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:45.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.851+0000 7fee917fa640 10 client.?.objecter ms_handle_connect 0x7fee9413f710 2024-03-25T19:36:45.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.851+0000 7fee917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:45.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.859+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854636426 0 0) 0x7fee94138b50 con 0x7fee940a98f0 2024-03-25T19:36:45.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.859+0000 7fee917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:45.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.859+0000 7fee917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:45.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.859+0000 7fee917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:45.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.859+0000 7fee917fa640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:36:45.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.859+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee78004eb0 con 0x7fee940a98f0 2024-03-25T19:36:45.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.860+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691282226 0 0) 0x7fee941394a0 con 0x7fee9413f710 2024-03-25T19:36:45.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.860+0000 7fee917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:45.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.860+0000 7fee917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:45.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.860+0000 7fee917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:45.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.860+0000 7fee917fa640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:36:45.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.860+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee94138b50 con 0x7fee9413f710 2024-03-25T19:36:45.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.882+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1171218803 0 0) 0x7fee78004eb0 con 0x7fee940a98f0 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.882+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee941394a0 con 0x7fee940a98f0 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.882+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee88001a30 con 0x7fee940a98f0 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.882+0000 7fee917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.882+0000 7fee917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.882+0000 7fee917fa640 10 monclient(hunting): dump: 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:45.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.882+0000 7fee917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:45.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.891+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502555259 0 0) 0x7fee94138b50 con 0x7fee9413f710 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.892+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee78004eb0 con 0x7fee9413f710 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.892+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee800018b0 con 0x7fee9413f710 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.892+0000 7fee917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.892+0000 7fee917fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.892+0000 7fee917fa640 10 monclient(hunting): dump: 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:45.892 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:45.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2729870179 0 0) 0x7fee941394a0 con 0x7fee940a98f0 2024-03-25T19:36:45.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 >> v1:172.21.15.159:6789/0 conn(0x7fee9413f710 legacy=0x7fee941d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 >> v1:172.21.15.134:6789/0 conn(0x7fee941430c0 legacy=0x7fee94134110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 1 monclient: found mon.c 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee941d8380 con 0x7fee940a98f0 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 10 monclient: _renew_subs 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:45.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee94139710 con 0x7fee940a98f0 2024-03-25T19:36:45.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:45.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 10 monclient: _renew_subs 2024-03-25T19:36:45.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:45.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee941d7750 con 0x7fee940a98f0 2024-03-25T19:36:45.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.902+0000 7fee9aab3640 1 librados: init done 2024-03-25T19:36:45.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.903+0000 7fee9aab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:45.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.903+0000 7fee9aab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:45.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.903+0000 7fee9aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:45.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:45.903+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee940627b0 con 0x7fee940a98f0 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.204+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee88001c70 con 0x7fee940a98f0 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.204+0000 7fee917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.204+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fee88003a60 con 0x7fee940a98f0 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.204+0000 7fee917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.204+0000 7fee917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.204+0000 7fee917fa640 10 monclient: dump: 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:46.204 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:46.205 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:46.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.204+0000 7fee92ffd640 4 set_mon_vals no callback set 2024-03-25T19:36:46.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.205+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fee88003c40 con 0x7fee940a98f0 2024-03-25T19:36:46.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.205+0000 7fee93fff640 10 monclient: get_auth_request con 0x7fee7806de90 auth_method 0 2024-03-25T19:36:46.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.206+0000 7fee917fa640 10 client.4589.objecter ms_handle_connect 0x7fee7806de90 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fee880548e0 con 0x7fee940a98f0 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 10 client.4589.objecter ms_dispatch 0x7fee94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee880503f0 con 0x7fee940a98f0 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:46.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.212+0000 7fee917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:46.281 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:50.953 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.287 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:50.960 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.317 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:50.990 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.334 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:51.006 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.359 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:51.032 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.387 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:51.060 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.426+0000 7fee9aab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:46.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.426+0000 7fee9aab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:46.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.426+0000 7fee9aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:46.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.426+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee94069f40 con 0x7fee940a98f0 2024-03-25T19:36:46.450 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:51.097 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.452+0000 7fee917fa640 1 -- 172.21.15.134:0/1691307299 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fee88052320 con 0x7fee940a98f0 2024-03-25T19:36:46.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.452+0000 7fee917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:46.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.452+0000 7fee917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:46.452 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:46.452 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 10 librados: watch_flush enter 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 10 librados: watch_flush exit 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 >> v1:172.21.15.159:6800/1875874887 conn(0x7fee7806de90 legacy=0x7fee78070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 10 monclient: shutdown 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 >> v1:172.21.15.134:6790/0 conn(0x7fee940a98f0 legacy=0x7fee94138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.454+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 shutdown_connections 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.455+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 >> 172.21.15.134:0/1691307299 conn(0x7fee940b7ea0 msgr2=0x7fee9413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.455+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 shutdown_connections 2024-03-25T19:36:46.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.455+0000 7fee9aab3640 1 -- 172.21.15.134:0/1691307299 wait complete. 2024-03-25T19:36:46.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:46.455+0000 7fee9aab3640 1 librados: shutdown 2024-03-25T19:36:46.456 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:51.102 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:46.489 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:51.136 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:53.273 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:53.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.375+0000 7fa8ab5f4640 10 monclient: get_monmap_and_config 2024-03-25T19:36:53.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.375+0000 7fa8ab5f4640 10 monclient: build_initial_monmap 2024-03-25T19:36:53.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.375+0000 7fa8ab5f4640 10 monclient: monmap: 2024-03-25T19:36:53.375 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:53.375 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:53.375 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:36:53.376824+0000 2024-03-25T19:36:53.375 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:36:53.376824+0000 2024-03-25T19:36:53.376 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:53.376 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:53.376 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:53.376 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:36:53.376 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:36:53.376 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.376+0000 7fa8ab5f4640 1 Processor -- start 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.376+0000 7fa8ab5f4640 1 -- start start 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.376+0000 7fa8ab5f4640 10 monclient: init 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 10 monclient(hunting): picked mon.noname-b con 0x7fa8a40b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a40b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:53.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a414be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:53.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40af940 con 0x7fa8a414be90 2024-03-25T19:36:53.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40b0a70 con 0x7fa8a40b1730 2024-03-25T19:36:53.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a40b0d50 con 0x7fa8a40b4d60 2024-03-25T19:36:53.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:53.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.377+0000 7fa8ab5f4640 10 monclient(hunting): authenticate will time out at 6445.535842s 2024-03-25T19:36:53.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.385+0000 7fa8a8b8f640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa8a40b1730 0x7fa8a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:33862/0 (socket says 172.21.15.134:33862) 2024-03-25T19:36:53.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.385+0000 7fa8a8b8f640 1 -- 172.21.15.134:0/3827671495 learned_addr learned my addr 172.21.15.134:0/3827671495 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.415+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748965427 0 0) 0x7fa8a40b0a70 con 0x7fa8a40b1730 2024-03-25T19:36:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.415+0000 7fa89bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.415+0000 7fa89bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.415+0000 7fa89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.415+0000 7fa89bfff640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:36:53.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.415+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa890003520 con 0x7fa8a40b1730 2024-03-25T19:36:53.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.430+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338669824 0 0) 0x7fa8a40b0d50 con 0x7fa8a40b4d60 2024-03-25T19:36:53.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.430+0000 7fa89bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:53.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.430+0000 7fa89bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:53.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.430+0000 7fa89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:53.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.430+0000 7fa89bfff640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:36:53.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.430+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a40b0a70 con 0x7fa8a40b4d60 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.439+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859680556 0 0) 0x7fa890003520 con 0x7fa8a40b1730 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 >> v1:172.21.15.134:6790/0 conn(0x7fa8a40b4d60 legacy=0x7fa8a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 >> v1:172.21.15.134:6789/0 conn(0x7fa8a414be90 legacy=0x7fa8a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 1 monclient: found mon.noname-c 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a40b1030 con 0x7fa8a40b1730 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 10 monclient: _finish_auth 0 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:53.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa894001690 con 0x7fa8a40b1730 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa89bfff640 10 monclient: dump: 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa8ab5f4640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:36:53.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.440+0000 7fa8ab5f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8940017f0 con 0x7fa8a40b1730 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa89bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa89bfff640 1 -- 172.21.15.134:0/3827671495 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa894001c00 con 0x7fa8a40b1730 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa8ab5f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa8ab5f4640 4 set_mon_vals no callback set 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa89bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa89bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.474+0000 7fa89bfff640 10 monclient: dump: 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.474 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:53.475 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:53.475 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:53.475 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:53.475 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:53.475 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.475+0000 7fa8ab5f4640 10 monclient: shutdown 2024-03-25T19:36:53.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.475+0000 7fa8ab5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:53.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.475+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/3827671495 >> v1:172.21.15.159:6789/0 conn(0x7fa8a40b1730 legacy=0x7fa8a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:53.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.475+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/3827671495 shutdown_connections 2024-03-25T19:36:53.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.475+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/3827671495 >> 172.21.15.134:0/3827671495 conn(0x7fa8a40617a0 msgr2=0x7fa8a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:53.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.475+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/3827671495 shutdown_connections 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.475+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/3827671495 wait complete. 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 10 monclient: build_initial_monmap 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 10 monclient: monmap: 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 1 librados: starting msgr at 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 1 librados: starting objecter 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 1 Processor -- start 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 1 -- start start 2024-03-25T19:36:53.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 1 librados: setting wanted keys 2024-03-25T19:36:53.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 1 librados: calling monclient init 2024-03-25T19:36:53.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.476+0000 7fa8ab5f4640 10 monclient: init 2024-03-25T19:36:53.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:53.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a40b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a414be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 10 monclient(hunting): picked mon.noname-b con 0x7fa8a413b6a0 addr v1:172.21.15.159:6789/0 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a41d5630 con 0x7fa8a40b4d60 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a41425b0 con 0x7fa8a413b6a0 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a41daf30 con 0x7fa8a414be90 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:53.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.477+0000 7fa8ab5f4640 10 monclient(hunting): authenticate will time out at 6445.636365s 2024-03-25T19:36:53.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.487+0000 7fa8a9390640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa8a40b4d60 0x7fa8a4140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49976/0 (socket says 172.21.15.134:49976) 2024-03-25T19:36:53.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.487+0000 7fa8a9390640 1 -- 172.21.15.134:0/4084813871 learned_addr learned my addr 172.21.15.134:0/4084813871 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:36:53.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.490+0000 7fa899ffb640 10 client.?.objecter ms_handle_connect 0x7fa8a40b4d60 2024-03-25T19:36:53.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.490+0000 7fa899ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:53.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.492+0000 7fa899ffb640 10 client.?.objecter ms_handle_connect 0x7fa8a414be90 2024-03-25T19:36:53.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.492+0000 7fa899ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:53.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.500+0000 7fa899ffb640 10 client.?.objecter ms_handle_connect 0x7fa8a413b6a0 2024-03-25T19:36:53.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.500+0000 7fa899ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:53.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.501+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204112183 0 0) 0x7fa8a41d5630 con 0x7fa8a40b4d60 2024-03-25T19:36:53.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.501+0000 7fa899ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:53.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.501+0000 7fa899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:53.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.501+0000 7fa899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:53.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.501+0000 7fa899ffb640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:36:53.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.501+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa888004eb0 con 0x7fa8a40b4d60 2024-03-25T19:36:53.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198467759 0 0) 0x7fa8a41425b0 con 0x7fa8a413b6a0 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a41d5630 con 0x7fa8a413b6a0 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071792651 0 0) 0x7fa8a41daf30 con 0x7fa8a414be90 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:53.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:53.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:36:53.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.507+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a41425b0 con 0x7fa8a414be90 2024-03-25T19:36:53.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.515+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1169445001 0 0) 0x7fa888004eb0 con 0x7fa8a40b4d60 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.515+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a41daf30 con 0x7fa8a40b4d60 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.515+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa89c001a10 con 0x7fa8a40b4d60 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.515+0000 7fa899ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.515+0000 7fa899ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.515+0000 7fa899ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.515+0000 7fa899ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.521+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 885812812 0 0) 0x7fa8a41425b0 con 0x7fa8a414be90 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.521+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa888004eb0 con 0x7fa8a414be90 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.521+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa894001b00 con 0x7fa8a414be90 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.521+0000 7fa899ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.521+0000 7fa899ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.521+0000 7fa899ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:53.521 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.523+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2757731665 0 0) 0x7fa8a41d5630 con 0x7fa8a413b6a0 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.524+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a41425b0 con 0x7fa8a413b6a0 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.524+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa8a0001b10 con 0x7fa8a413b6a0 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.524+0000 7fa899ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.524+0000 7fa899ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.524+0000 7fa899ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:53.524 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2069932120 0 0) 0x7fa888004eb0 con 0x7fa8a414be90 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 >> v1:172.21.15.159:6789/0 conn(0x7fa8a413b6a0 legacy=0x7fa8a41db500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 >> v1:172.21.15.134:6789/0 conn(0x7fa8a40b4d60 legacy=0x7fa8a4140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 1 monclient: found mon.c 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a41db0c0 con 0x7fa8a414be90 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 5 monclient: authenticate success, global_id 4595 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 10 monclient: _renew_subs 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:53.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a413bc30 con 0x7fa8a414be90 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 10 monclient: _renew_subs 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a41433c0 con 0x7fa8a414be90 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.534+0000 7fa8ab5f4640 1 librados: init done 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.535+0000 7fa8ab5f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.535+0000 7fa8ab5f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:53.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.535+0000 7fa8ab5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:53.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.535+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a41423a0 con 0x7fa8a414be90 2024-03-25T19:36:53.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.840+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa894001240 con 0x7fa8a414be90 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.840+0000 7fa899ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.840+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fa894004a90 con 0x7fa8a414be90 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.840+0000 7fa899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.840+0000 7fa899ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.840+0000 7fa899ffb640 10 monclient: dump: 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:36:53.840 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:36:53.841 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:36:53.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.840+0000 7fa89b7fe640 4 set_mon_vals no callback set 2024-03-25T19:36:53.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.841+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fa89401aaa0 con 0x7fa8a414be90 2024-03-25T19:36:53.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.841+0000 7fa8a9390640 10 monclient: get_auth_request con 0x7fa88806e160 auth_method 0 2024-03-25T19:36:53.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.841+0000 7fa899ffb640 10 client.4595.objecter ms_handle_connect 0x7fa88806e160 2024-03-25T19:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fa894054660 con 0x7fa8a414be90 2024-03-25T19:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 10 client.4595.objecter ms_dispatch 0x7fa8a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:53.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T19:36:53.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa894050170 con 0x7fa8a414be90 2024-03-25T19:36:53.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:53.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:53.847+0000 7fa899ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:54.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.101+0000 7fa8ab5f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:54.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.101+0000 7fa8ab5f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:54.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.101+0000 7fa8ab5f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:36:54.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.101+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8a40627b0 con 0x7fa8a414be90 2024-03-25T19:36:54.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.124+0000 7fa899ffb640 1 -- 172.21.15.134:0/4084813871 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fa8940520a0 con 0x7fa8a414be90 2024-03-25T19:36:54.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.124+0000 7fa899ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:54.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.124+0000 7fa899ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:54.124 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:36:54.124 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:54.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 10 librados: watch_flush enter 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 10 librados: watch_flush exit 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 10 client.4595.objecter successfully canceled tick 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 >> v1:172.21.15.159:6800/1875874887 conn(0x7fa88806e160 legacy=0x7fa888070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 10 monclient: shutdown 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 >> v1:172.21.15.134:6790/0 conn(0x7fa8a414be90 legacy=0x7fa8a4142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 shutdown_connections 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 >> 172.21.15.134:0/4084813871 conn(0x7fa8a40617a0 msgr2=0x7fa8a40b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 shutdown_connections 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.127+0000 7fa8ab5f4640 1 -- 172.21.15.134:0/4084813871 wait complete. 2024-03-25T19:36:54.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:36:54.128+0000 7fa8ab5f4640 1 librados: shutdown 2024-03-25T19:36:54.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:00.945 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:01.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.059+0000 7f4c90c31640 10 monclient: get_monmap_and_config 2024-03-25T19:37:01.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.059+0000 7f4c90c31640 10 monclient: build_initial_monmap 2024-03-25T19:37:01.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.059+0000 7f4c90c31640 10 monclient: monmap: 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:01.060887+0000 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:01.060887+0000 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:01.060 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.060+0000 7f4c90c31640 1 Processor -- start 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.060+0000 7f4c90c31640 1 -- start start 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.060+0000 7f4c90c31640 10 monclient: init 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 10 monclient(hunting): picked mon.noname-c con 0x7f4c8c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 10 monclient(hunting): picked mon.noname-a con 0x7f4c8c0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 10 monclient(hunting): picked mon.noname-b con 0x7f4c8c14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c0af940 con 0x7f4c8c0b1730 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c0b0a70 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c0b0d50 con 0x7f4c8c14be90 2024-03-25T19:37:01.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:01.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.061+0000 7f4c90c31640 10 monclient(hunting): authenticate will time out at 6453.219851s 2024-03-25T19:37:01.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.074+0000 7f4c8a59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f4c8c0b4d60 0x7f4c8c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:59780/0 (socket says 172.21.15.134:59780) 2024-03-25T19:37:01.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.074+0000 7f4c8a59c640 1 -- 172.21.15.134:0/3450053750 learned_addr learned my addr 172.21.15.134:0/3450053750 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:01.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.087+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179692941 0 0) 0x7f4c8c0b0a70 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.087+0000 7f4c89d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:01.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.087+0000 7f4c89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:01.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.087+0000 7f4c89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:01.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.087+0000 7f4c89d9b640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:37:01.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.087+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c78003520 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.089+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578344261 0 0) 0x7f4c8c0b0d50 con 0x7f4c8c14be90 2024-03-25T19:37:01.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.089+0000 7f4c89d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:01.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.089+0000 7f4c89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:01.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.089+0000 7f4c89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:01.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.089+0000 7f4c89d9b640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:37:01.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.089+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c0b0a70 con 0x7f4c8c14be90 2024-03-25T19:37:01.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.104+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898780934 0 0) 0x7f4c8c0af940 con 0x7f4c8c0b1730 2024-03-25T19:37:01.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.104+0000 7f4c89d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:01.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.104+0000 7f4c89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:01.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.104+0000 7f4c89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:01.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.104+0000 7f4c89d9b640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:37:01.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.104+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c0b0d50 con 0x7f4c8c0b1730 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4224562279 0 0) 0x7f4c78003520 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 >> v1:172.21.15.134:6790/0 conn(0x7f4c8c14be90 legacy=0x7f4c8c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 >> v1:172.21.15.134:6789/0 conn(0x7f4c8c0b1730 legacy=0x7f4c8c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 1 monclient: found mon.noname-c 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c0b1030 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 10 monclient: _finish_auth 0 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:37:01.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4c7c001690 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c89d9b640 10 monclient: dump: 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c90c31640 5 monclient: authenticate success, global_id 4591 2024-03-25T19:37:01.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.112+0000 7f4c90c31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c7c0017f0 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c89d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c89d9b640 1 -- 172.21.15.134:0/3450053750 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4c7c001c00 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c89d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c89d9b640 10 monclient: dump: 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:01.131 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:01.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c90c31640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:01.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.131+0000 7f4c90c31640 4 set_mon_vals no callback set 2024-03-25T19:37:01.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.132+0000 7f4c90c31640 10 monclient: shutdown 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.132+0000 7f4c90c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.132+0000 7f4c90c31640 1 -- 172.21.15.134:0/3450053750 >> v1:172.21.15.159:6789/0 conn(0x7f4c8c0b4d60 legacy=0x7f4c8c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.132+0000 7f4c90c31640 1 -- 172.21.15.134:0/3450053750 shutdown_connections 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.132+0000 7f4c90c31640 1 -- 172.21.15.134:0/3450053750 >> 172.21.15.134:0/3450053750 conn(0x7f4c8c0617a0 msgr2=0x7f4c8c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.132+0000 7f4c90c31640 1 -- 172.21.15.134:0/3450053750 shutdown_connections 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.132+0000 7f4c90c31640 1 -- 172.21.15.134:0/3450053750 wait complete. 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 10 monclient: build_initial_monmap 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 10 monclient: monmap: 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:01.133 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:01.134 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:01.134 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:01.134 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:01.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 1 librados: starting msgr at 2024-03-25T19:37:01.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 1 librados: starting objecter 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 1 Processor -- start 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 1 -- start start 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 1 librados: setting wanted keys 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 1 librados: calling monclient init 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.133+0000 7f4c90c31640 10 monclient: init 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 10 monclient(hunting): picked mon.noname-c con 0x7f4c8c0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 10 monclient(hunting): picked mon.noname-b con 0x7f4c8c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 10 monclient(hunting): picked mon.noname-a con 0x7f4c8c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c141ec0 con 0x7f4c8c14be90 2024-03-25T19:37:01.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c142110 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c1d79d0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:01.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.134+0000 7f4c90c31640 10 monclient(hunting): authenticate will time out at 6453.293230s 2024-03-25T19:37:01.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.147+0000 7f4c8ad9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4c8c14be90 0x7f4c8c142d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:33752/0 (socket says 172.21.15.134:33752) 2024-03-25T19:37:01.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.147+0000 7f4c8ad9d640 1 -- 172.21.15.134:0/2725688178 learned_addr learned my addr 172.21.15.134:0/2725688178 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:01.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.147+0000 7f4c837fe640 10 client.?.objecter ms_handle_connect 0x7f4c8c14be90 2024-03-25T19:37:01.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.147+0000 7f4c837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:01.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.153+0000 7f4c837fe640 10 client.?.objecter ms_handle_connect 0x7f4c8c0b1730 2024-03-25T19:37:01.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.153+0000 7f4c837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:01.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.156+0000 7f4c837fe640 10 client.?.objecter ms_handle_connect 0x7f4c8c0b4d60 2024-03-25T19:37:01.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.156+0000 7f4c837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:01.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.161+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917916536 0 0) 0x7f4c8c1d79d0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.161+0000 7f4c837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:01.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.161+0000 7f4c837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:01.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.161+0000 7f4c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:01.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.161+0000 7f4c837fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:37:01.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.161+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c6c004eb0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.167+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868992652 0 0) 0x7f4c8c141ec0 con 0x7f4c8c14be90 2024-03-25T19:37:01.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.167+0000 7f4c837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:01.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.167+0000 7f4c837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:01.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.167+0000 7f4c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:01.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.167+0000 7f4c837fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:37:01.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.167+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c1d79d0 con 0x7f4c8c14be90 2024-03-25T19:37:01.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.171+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719206023 0 0) 0x7f4c8c142110 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.171+0000 7f4c837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:01.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.171+0000 7f4c837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:01.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.171+0000 7f4c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:01.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.171+0000 7f4c837fe640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:37:01.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.171+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c141ec0 con 0x7f4c8c0b4d60 2024-03-25T19:37:01.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.177+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1935146011 0 0) 0x7f4c6c004eb0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.177+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c8c142110 con 0x7f4c8c0b1730 2024-03-25T19:37:01.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.177+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4c7c001b00 con 0x7f4c8c0b1730 2024-03-25T19:37:01.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.177+0000 7f4c837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.177+0000 7f4c837fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.177+0000 7f4c837fe640 10 monclient(hunting): dump: 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:01.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.177+0000 7f4c837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:01.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.182+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3911935450 0 0) 0x7f4c8c1d79d0 con 0x7f4c8c14be90 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.182+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c6c004eb0 con 0x7f4c8c14be90 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.182+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4c84001c10 con 0x7f4c8c14be90 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.182+0000 7f4c837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.182+0000 7f4c837fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.182+0000 7f4c837fe640 10 monclient(hunting): dump: 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:01.183 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2700856359 0 0) 0x7f4c8c142110 con 0x7f4c8c0b1730 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 >> v1:172.21.15.159:6789/0 conn(0x7f4c8c0b4d60 legacy=0x7f4c8c1d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 >> v1:172.21.15.134:6789/0 conn(0x7f4c8c14be90 legacy=0x7f4c8c142d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 1 monclient: found mon.c 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c1d8600 con 0x7f4c8c0b1730 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c90c31640 5 monclient: authenticate success, global_id 4601 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c90c31640 10 monclient: _renew_subs 2024-03-25T19:37:01.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.190+0000 7f4c90c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.191+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c143450 con 0x7f4c8c0b1730 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.191+0000 7f4c90c31640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.191+0000 7f4c90c31640 10 monclient: _renew_subs 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.191+0000 7f4c90c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.191+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c13bb10 con 0x7f4c8c0b1730 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.191+0000 7f4c90c31640 1 librados: init done 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.192+0000 7f4c90c31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.192+0000 7f4c90c31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:01.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.192+0000 7f4c90c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:01.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.192+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c0627b0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c7c001220 con 0x7f4c8c0b1730 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f4c7c004c30 con 0x7f4c8c0b1730 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c837fe640 10 monclient: dump: 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.488 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c8959a640 4 set_mon_vals no callback set 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.488+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f4c7c01ac40 con 0x7f4c8c0b1730 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.489+0000 7f4c81d9b640 10 monclient: get_auth_request con 0x7f4c6c06df60 auth_method 0 2024-03-25T19:37:01.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.489+0000 7f4c837fe640 10 client.4601.objecter ms_handle_connect 0x7f4c6c06df60 2024-03-25T19:37:01.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f4c7c0548b0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 10 client.4601.objecter ms_dispatch 0x7f4c8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:01.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:01.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:01.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T19:37:01.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c7c0501f0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:01.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.496+0000 7f4c837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:01.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.726+0000 7f4c90c31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:01.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.726+0000 7f4c90c31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:01.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.726+0000 7f4c90c31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:01.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.726+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c8c1d7b60 con 0x7f4c8c0b1730 2024-03-25T19:37:01.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.757+0000 7f4c837fe640 1 -- 172.21.15.134:0/2725688178 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f4c7c0522f0 con 0x7f4c8c0b1730 2024-03-25T19:37:01.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.757+0000 7f4c837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:01.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.757+0000 7f4c837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:01.757 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:01.757 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.759+0000 7f4c90c31640 10 librados: watch_flush enter 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.759+0000 7f4c90c31640 10 librados: watch_flush exit 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.759+0000 7f4c90c31640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.759+0000 7f4c90c31640 10 client.4601.objecter successfully canceled tick 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 >> v1:172.21.15.159:6800/1875874887 conn(0x7f4c6c06df60 legacy=0x7f4c6c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 10 monclient: shutdown 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 >> v1:172.21.15.134:6790/0 conn(0x7f4c8c0b1730 legacy=0x7f4c8c140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 shutdown_connections 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 >> 172.21.15.134:0/2725688178 conn(0x7f4c8c0617a0 msgr2=0x7f4c8c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 shutdown_connections 2024-03-25T19:37:01.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 1 -- 172.21.15.134:0/2725688178 wait complete. 2024-03-25T19:37:01.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:01.760+0000 7f4c90c31640 1 librados: shutdown 2024-03-25T19:37:01.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:06.143 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:10.815 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.156 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:10.829 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.185 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:10.858 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.199 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:10.846 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.206 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:10.853 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.234 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:10.907 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.246 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:10.918 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.246 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:10.893 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:06.284 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:10.956 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:08.576 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.697+0000 7f9dfbe33640 10 monclient: get_monmap_and_config 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.697+0000 7f9dfbe33640 10 monclient: build_initial_monmap 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.697+0000 7f9dfbe33640 10 monclient: monmap: 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:08.698327+0000 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:08.698327+0000 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:08.697 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:08.698 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:08.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.698+0000 7f9dfbe33640 1 Processor -- start 2024-03-25T19:37:08.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.698+0000 7f9dfbe33640 1 -- start start 2024-03-25T19:37:08.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.698+0000 7f9dfbe33640 10 monclient: init 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 10 monclient(hunting): picked mon.noname-a con 0x7f9df40ae6e0 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 10 monclient(hunting): picked mon.noname-b con 0x7f9df406f080 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 10 monclient(hunting): picked mon.noname-c con 0x7f9df40704e0 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df40718d0 con 0x7f9df40ae6e0 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df406fa30 con 0x7f9df40704e0 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df4070090 con 0x7f9df406f080 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:08.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.699+0000 7f9dfbe33640 10 monclient(hunting): authenticate will time out at 6460.858062s 2024-03-25T19:37:08.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.705+0000 7f9dfa3d0640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9df40704e0 0x7f9df414a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:36330/0 (socket says 172.21.15.134:36330) 2024-03-25T19:37:08.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.706+0000 7f9dfa3d0640 1 -- 172.21.15.134:0/2292217409 learned_addr learned my addr 172.21.15.134:0/2292217409 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:08.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.715+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526932534 0 0) 0x7f9df406fa30 con 0x7f9df40704e0 2024-03-25T19:37:08.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.715+0000 7f9df8bcd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:08.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.715+0000 7f9df8bcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:08.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.715+0000 7f9df8bcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:08.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.715+0000 7f9df8bcd640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:37:08.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.715+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de0003520 con 0x7f9df40704e0 2024-03-25T19:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.721+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823925346 0 0) 0x7f9df4070090 con 0x7f9df406f080 2024-03-25T19:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.721+0000 7f9df8bcd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.721+0000 7f9df8bcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.721+0000 7f9df8bcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.721+0000 7f9df8bcd640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.721+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df406fa30 con 0x7f9df406f080 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332543672 0 0) 0x7f9de0003520 con 0x7f9df40704e0 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 >> v1:172.21.15.134:6790/0 conn(0x7f9df406f080 legacy=0x7f9df4071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 >> v1:172.21.15.134:6789/0 conn(0x7f9df40ae6e0 legacy=0x7f9df413c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 1 monclient: found mon.noname-c 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df406f510 con 0x7f9df40704e0 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 10 monclient: _finish_auth 0 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9dec001a10 con 0x7f9df40704e0 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9dfbe33640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:37:08.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9dfbe33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9df8bcd640 10 monclient: dump: 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:08.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.728+0000 7f9dfbe33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:08.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.735+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dec001b70 con 0x7f9df40704e0 2024-03-25T19:37:08.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.735+0000 7f9df8bcd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:08.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.735+0000 7f9dfbe33640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:08.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.735+0000 7f9dfbe33640 4 set_mon_vals no callback set 2024-03-25T19:37:08.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9dfbe33640 10 monclient: shutdown 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9dfbe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2292217409 >> v1:172.21.15.159:6789/0 conn(0x7f9df40704e0 legacy=0x7f9df414a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9df8bcd640 1 -- 172.21.15.134:0/2292217409 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9dec001f80 con 0x7f9df40704e0 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9df8bcd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2292217409 shutdown_connections 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2292217409 >> 172.21.15.134:0/2292217409 conn(0x7f9df406b630 msgr2=0x7f9df40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2292217409 shutdown_connections 2024-03-25T19:37:08.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.736+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2292217409 wait complete. 2024-03-25T19:37:08.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.737+0000 7f9dfbe33640 10 monclient: build_initial_monmap 2024-03-25T19:37:08.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.737+0000 7f9dfbe33640 10 monclient: monmap: 2024-03-25T19:37:08.738 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.737+0000 7f9dfbe33640 1 librados: starting msgr at 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.737+0000 7f9dfbe33640 1 librados: starting objecter 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.737+0000 7f9dfbe33640 1 Processor -- start 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 1 -- start start 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 1 librados: setting wanted keys 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 1 librados: calling monclient init 2024-03-25T19:37:08.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient: init 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient(hunting): picked mon.noname-b con 0x7f9df40704e0 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient(hunting): picked mon.noname-c con 0x7f9df40ae6e0 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient(hunting): picked mon.noname-a con 0x7f9df41431f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df41413a0 con 0x7f9df41431f0 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df4141680 con 0x7f9df40704e0 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df4140370 con 0x7f9df40ae6e0 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:08.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.738+0000 7f9dfbe33640 10 monclient(hunting): authenticate will time out at 6460.897246s 2024-03-25T19:37:08.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.744+0000 7f9df9bcf640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9df40704e0 0x7f9df4141e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:36362/0 (socket says 172.21.15.134:36362) 2024-03-25T19:37:08.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.744+0000 7f9df9bcf640 1 -- 172.21.15.134:0/2818477875 learned_addr learned my addr 172.21.15.134:0/2818477875 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:08.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.744+0000 7f9dea7fc640 10 client.?.objecter ms_handle_connect 0x7f9df40704e0 2024-03-25T19:37:08.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.745+0000 7f9dea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:08.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.748+0000 7f9dea7fc640 10 client.?.objecter ms_handle_connect 0x7f9df40ae6e0 2024-03-25T19:37:08.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.748+0000 7f9dea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:08.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.756+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177463207 0 0) 0x7f9df4140370 con 0x7f9df40ae6e0 2024-03-25T19:37:08.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.756+0000 7f9dea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:08.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.756+0000 7f9dea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:08.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.756+0000 7f9dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:08.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.756+0000 7f9dea7fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:37:08.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.756+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd8004eb0 con 0x7f9df40ae6e0 2024-03-25T19:37:08.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.758+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102888855 0 0) 0x7f9df4141680 con 0x7f9df40704e0 2024-03-25T19:37:08.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.758+0000 7f9dea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:08.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.758+0000 7f9dea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:08.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.758+0000 7f9dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:08.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.758+0000 7f9dea7fc640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:37:08.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.758+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df4140370 con 0x7f9df40704e0 2024-03-25T19:37:08.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.760+0000 7f9dea7fc640 10 client.?.objecter ms_handle_connect 0x7f9df41431f0 2024-03-25T19:37:08.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.760+0000 7f9dea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:08.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.766+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028870441 0 0) 0x7f9df41413a0 con 0x7f9df41431f0 2024-03-25T19:37:08.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.766+0000 7f9dea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:08.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.766+0000 7f9dea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:08.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.766+0000 7f9dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:08.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.766+0000 7f9dea7fc640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:37:08.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.766+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df4141680 con 0x7f9df41431f0 2024-03-25T19:37:08.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.769+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2394665852 0 0) 0x7f9dd8004eb0 con 0x7f9df40ae6e0 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.769+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9df41413a0 con 0x7f9df40ae6e0 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.769+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9de4001930 con 0x7f9df40ae6e0 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.769+0000 7f9dea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.769+0000 7f9dea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.769+0000 7f9dea7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:08.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.769+0000 7f9dea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:08.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2240151771 0 0) 0x7f9df41413a0 con 0x7f9df40ae6e0 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 >> v1:172.21.15.159:6789/0 conn(0x7f9df40704e0 legacy=0x7f9df4141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 >> v1:172.21.15.134:6789/0 conn(0x7f9df41431f0 legacy=0x7f9df41e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 1 monclient: found mon.c 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4140650 con 0x7f9df40ae6e0 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 5 monclient: authenticate success, global_id 4607 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 10 monclient: _renew_subs 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df4150cf0 con 0x7f9df40ae6e0 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:08.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 10 monclient: _renew_subs 2024-03-25T19:37:08.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:08.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df413eac0 con 0x7f9df40ae6e0 2024-03-25T19:37:08.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.776+0000 7f9dfbe33640 1 librados: init done 2024-03-25T19:37:08.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.777+0000 7f9dfbe33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:08.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.777+0000 7f9dfbe33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:08.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.777+0000 7f9dfbe33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:08.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:08.777+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df40627b0 con 0x7f9df40ae6e0 2024-03-25T19:37:09.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.068+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de4003610 con 0x7f9df40ae6e0 2024-03-25T19:37:09.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.068+0000 7f9dea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:09.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.068+0000 7f9debfff640 4 set_mon_vals no callback set 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.077+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f9de4003920 con 0x7f9df40ae6e0 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.077+0000 7f9dea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.077+0000 7f9dea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.077+0000 7f9dea7fc640 10 monclient: dump: 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:09.077 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.078+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f9de40198f0 con 0x7f9df40ae6e0 2024-03-25T19:37:09.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.078+0000 7f9df9bcf640 10 monclient: get_auth_request con 0x7f9dd806dda0 auth_method 0 2024-03-25T19:37:09.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.079+0000 7f9dea7fc640 10 client.4607.objecter ms_handle_connect 0x7f9dd806dda0 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f9de40536b0 con 0x7f9df40ae6e0 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 10 client.4607.objecter ms_dispatch 0x7f9df4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9de404eff0 con 0x7f9df40ae6e0 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:09.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.084+0000 7f9dea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:09.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.343+0000 7f9dfbe33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:09.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.343+0000 7f9dfbe33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:09.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.343+0000 7f9dfbe33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:09.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.343+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9df40681e0 con 0x7f9df40ae6e0 2024-03-25T19:37:09.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.380+0000 7f9dea7fc640 1 -- 172.21.15.134:0/2818477875 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f9de40510f0 con 0x7f9df40ae6e0 2024-03-25T19:37:09.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.380+0000 7f9dea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:09.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.380+0000 7f9dea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:09.380 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:09.380 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:09.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 10 librados: watch_flush enter 2024-03-25T19:37:09.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 10 librados: watch_flush exit 2024-03-25T19:37:09.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 10 client.4607.objecter successfully canceled tick 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 >> v1:172.21.15.159:6800/1875874887 conn(0x7f9dd806dda0 legacy=0x7f9dd8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 10 monclient: shutdown 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 >> v1:172.21.15.134:6790/0 conn(0x7f9df40ae6e0 legacy=0x7f9df41deff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 shutdown_connections 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 >> 172.21.15.134:0/2818477875 conn(0x7f9df406b630 msgr2=0x7f9df413c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 shutdown_connections 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 1 -- 172.21.15.134:0/2818477875 wait complete. 2024-03-25T19:37:09.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:09.383+0000 7f9dfbe33640 1 librados: shutdown 2024-03-25T19:37:09.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:16.200 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.318+0000 7ffb7d44a640 10 monclient: get_monmap_and_config 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.318+0000 7ffb7d44a640 10 monclient: build_initial_monmap 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.318+0000 7ffb7d44a640 10 monclient: monmap: 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:16.319977+0000 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:16.319977+0000 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:16.319 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:16.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.319+0000 7ffb7d44a640 1 Processor -- start 2024-03-25T19:37:16.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.319+0000 7ffb7d44a640 1 -- start start 2024-03-25T19:37:16.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.319+0000 7ffb7d44a640 10 monclient: init 2024-03-25T19:37:16.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 10 monclient(hunting): picked mon.noname-b con 0x7ffb780a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 10 monclient(hunting): picked mon.noname-c con 0x7ffb7813f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb781430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb78142520 con 0x7ffb781430c0 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780b28a0 con 0x7ffb7813f710 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780b2a30 con 0x7ffb780a98f0 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:16.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.320+0000 7ffb7d44a640 10 monclient(hunting): authenticate will time out at 6468.479622s 2024-03-25T19:37:16.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.328+0000 7ffb6ffff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ffb7813f710 0x7ffb781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:45390/0 (socket says 172.21.15.134:45390) 2024-03-25T19:37:16.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.328+0000 7ffb6ffff640 1 -- 172.21.15.134:0/2689232829 learned_addr learned my addr 172.21.15.134:0/2689232829 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:16.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.337+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081880413 0 0) 0x7ffb780b28a0 con 0x7ffb7813f710 2024-03-25T19:37:16.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.337+0000 7ffb767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:16.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.337+0000 7ffb767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:16.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.337+0000 7ffb767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:16.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.337+0000 7ffb767fc640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:37:16.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.337+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64003520 con 0x7ffb7813f710 2024-03-25T19:37:16.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.341+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066420901 0 0) 0x7ffb780b2a30 con 0x7ffb780a98f0 2024-03-25T19:37:16.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.341+0000 7ffb767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:16.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.341+0000 7ffb767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:16.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.341+0000 7ffb767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:16.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.341+0000 7ffb767fc640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:37:16.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.341+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb780b28a0 con 0x7ffb780a98f0 2024-03-25T19:37:16.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1633684246 0 0) 0x7ffb64003520 con 0x7ffb7813f710 2024-03-25T19:37:16.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 >> v1:172.21.15.134:6790/0 conn(0x7ffb780a98f0 legacy=0x7ffb780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:16.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 >> v1:172.21.15.134:6789/0 conn(0x7ffb781430c0 legacy=0x7ffb7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:16.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:16.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 1 monclient: found mon.noname-c 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780b44c0 con 0x7ffb7813f710 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ffb60001690 con 0x7ffb7813f710 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb767fc640 10 monclient: dump: 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.346 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:16.347 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:16.347 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:16.347 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:16.347 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:16.347 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:16.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb7d44a640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:37:16.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.345+0000 7ffb7d44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb600017f0 con 0x7ffb7813f710 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb767fc640 1 -- 172.21.15.134:0/2689232829 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ffb60001c00 con 0x7ffb7813f710 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb767fc640 10 monclient: dump: 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:16.358 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb7d44a640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:16.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.358+0000 7ffb7d44a640 4 set_mon_vals no callback set 2024-03-25T19:37:16.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.359+0000 7ffb7d44a640 10 monclient: shutdown 2024-03-25T19:37:16.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.359+0000 7ffb7d44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.359+0000 7ffb7d44a640 1 -- 172.21.15.134:0/2689232829 >> v1:172.21.15.159:6789/0 conn(0x7ffb7813f710 legacy=0x7ffb781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.359+0000 7ffb7d44a640 1 -- 172.21.15.134:0/2689232829 shutdown_connections 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.359+0000 7ffb7d44a640 1 -- 172.21.15.134:0/2689232829 >> 172.21.15.134:0/2689232829 conn(0x7ffb780b7ea0 msgr2=0x7ffb780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.359+0000 7ffb7d44a640 1 -- 172.21.15.134:0/2689232829 shutdown_connections 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.359+0000 7ffb7d44a640 1 -- 172.21.15.134:0/2689232829 wait complete. 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.360+0000 7ffb7d44a640 10 monclient: build_initial_monmap 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.360+0000 7ffb7d44a640 10 monclient: monmap: 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:16.361 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.360+0000 7ffb7d44a640 1 librados: starting msgr at 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.360+0000 7ffb7d44a640 1 librados: starting objecter 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 1 Processor -- start 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 1 -- start start 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 1 librados: setting wanted keys 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 1 librados: calling monclient init 2024-03-25T19:37:16.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient: init 2024-03-25T19:37:16.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:16.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:16.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient(hunting): picked mon.noname-c con 0x7ffb780a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb7813f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient(hunting): picked mon.noname-b con 0x7ffb781430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb781d77a0 con 0x7ffb7813f710 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb78138e40 con 0x7ffb781430c0 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780aba30 con 0x7ffb780a98f0 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.361+0000 7ffb7d44a640 10 monclient(hunting): authenticate will time out at 6468.520538s 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.362+0000 7ffb76ffd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7ffb780a98f0 0x7ffb78138730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:58372/0 (socket says 172.21.15.134:58372) 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.362+0000 7ffb76ffd640 1 -- 172.21.15.134:0/4011153813 learned_addr learned my addr 172.21.15.134:0/4011153813 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.362+0000 7ffb6f7fe640 10 client.?.objecter ms_handle_connect 0x7ffb780a98f0 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.362+0000 7ffb6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.363+0000 7ffb6f7fe640 10 client.?.objecter ms_handle_connect 0x7ffb781430c0 2024-03-25T19:37:16.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.363+0000 7ffb6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:16.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.418+0000 7ffb6f7fe640 10 client.?.objecter ms_handle_connect 0x7ffb7813f710 2024-03-25T19:37:16.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.418+0000 7ffb6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:16.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.440+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413640315 0 0) 0x7ffb781d77a0 con 0x7ffb7813f710 2024-03-25T19:37:16.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.440+0000 7ffb6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:16.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.440+0000 7ffb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:16.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.440+0000 7ffb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:16.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.440+0000 7ffb6f7fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:37:16.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.440+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb58004eb0 con 0x7ffb7813f710 2024-03-25T19:37:16.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.459+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2087770504 0 0) 0x7ffb58004eb0 con 0x7ffb7813f710 2024-03-25T19:37:16.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.459+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb781d77a0 con 0x7ffb7813f710 2024-03-25T19:37:16.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.459+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ffb60001b00 con 0x7ffb7813f710 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.459+0000 7ffb6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.459+0000 7ffb6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.459+0000 7ffb6f7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:16.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.459+0000 7ffb6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2124536753 0 0) 0x7ffb781d77a0 con 0x7ffb7813f710 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 >> v1:172.21.15.134:6790/0 conn(0x7ffb780a98f0 legacy=0x7ffb78138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 >> v1:172.21.15.159:6789/0 conn(0x7ffb781430c0 legacy=0x7ffb78134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 1 monclient: found mon.a 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb781d8340 con 0x7ffb7813f710 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:16.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 5 monclient: authenticate success, global_id 4644 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 10 monclient: _renew_subs 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb78134820 con 0x7ffb7813f710 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 10 client.4644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 10 monclient: _renew_subs 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb781380c0 con 0x7ffb7813f710 2024-03-25T19:37:16.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.469+0000 7ffb7d44a640 1 librados: init done 2024-03-25T19:37:16.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.470+0000 7ffb7d44a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:16.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.470+0000 7ffb7d44a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:16.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.470+0000 7ffb7d44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:16.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.470+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb780627b0 con 0x7ffb7813f710 2024-03-25T19:37:16.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.757+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb60001220 con 0x7ffb7813f710 2024-03-25T19:37:16.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.757+0000 7ffb6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:16.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.757+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7ffb60004c30 con 0x7ffb7813f710 2024-03-25T19:37:16.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.757+0000 7ffb6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:16.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.757+0000 7ffb6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:16.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.757+0000 7ffb6f7fe640 10 monclient: dump: 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:16.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.757+0000 7ffb75ffb640 4 set_mon_vals no callback set 2024-03-25T19:37:16.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.758+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7ffb6001ac80 con 0x7ffb7813f710 2024-03-25T19:37:16.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.759+0000 7ffb76ffd640 10 monclient: get_auth_request con 0x7ffb5806dd30 auth_method 0 2024-03-25T19:37:16.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.759+0000 7ffb6f7fe640 10 client.4644.objecter ms_handle_connect 0x7ffb5806dd30 2024-03-25T19:37:16.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7ffb60054890 con 0x7ffb7813f710 2024-03-25T19:37:16.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 10 client.4644.objecter ms_dispatch 0x7ffb78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:16.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 3 client.4644.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:16.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 3 client.4644.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:16.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 20 client.4644.objecter dump_active .. 0 homeless 2024-03-25T19:37:16.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb600501d0 con 0x7ffb7813f710 2024-03-25T19:37:16.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:16.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:16.765+0000 7ffb6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:17.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.010+0000 7ffb7d44a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:17.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.010+0000 7ffb7d44a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:17.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.010+0000 7ffb7d44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:17.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.010+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb7814e8b0 con 0x7ffb7813f710 2024-03-25T19:37:17.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.045+0000 7ffb6f7fe640 1 -- 172.21.15.134:0/4011153813 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7ffb600522d0 con 0x7ffb7813f710 2024-03-25T19:37:17.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.045+0000 7ffb6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:17.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.045+0000 7ffb6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:17.046 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:17.046 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 10 librados: watch_flush enter 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 10 librados: watch_flush exit 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 20 client.4644.objecter shutdown clearing up homeless session... 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 10 client.4644.objecter successfully canceled tick 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 >> v1:172.21.15.159:6800/1875874887 conn(0x7ffb5806dd30 legacy=0x7ffb580701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 10 monclient: shutdown 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:17.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.048+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 >> v1:172.21.15.134:6789/0 conn(0x7ffb7813f710 legacy=0x7ffb781d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:17.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.049+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 shutdown_connections 2024-03-25T19:37:17.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.049+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 >> 172.21.15.134:0/4011153813 conn(0x7ffb780b7ea0 msgr2=0x7ffb78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:17.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.049+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 shutdown_connections 2024-03-25T19:37:17.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.049+0000 7ffb7d44a640 1 -- 172.21.15.134:0/4011153813 wait complete. 2024-03-25T19:37:17.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:17.049+0000 7ffb7d44a640 1 librados: shutdown 2024-03-25T19:37:17.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:23.868 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:23.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.983+0000 7f0f71389640 10 monclient: get_monmap_and_config 2024-03-25T19:37:23.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.983+0000 7f0f71389640 10 monclient: build_initial_monmap 2024-03-25T19:37:23.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.983+0000 7f0f71389640 10 monclient: monmap: 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:23.984125+0000 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:23.984125+0000 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:23.983 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:23.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.983+0000 7f0f71389640 1 Processor -- start 2024-03-25T19:37:23.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.983+0000 7f0f71389640 1 -- start start 2024-03-25T19:37:23.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.983+0000 7f0f71389640 10 monclient: init 2024-03-25T19:37:23.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 10 monclient(hunting): picked mon.noname-a con 0x7f0f6c0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 10 monclient(hunting): picked mon.noname-c con 0x7f0f6c13f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 10 monclient(hunting): picked mon.noname-b con 0x7f0f6c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c142520 con 0x7f0f6c0a98f0 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0b28a0 con 0x7f0f6c13f710 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0b2a30 con 0x7f0f6c1430c0 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:23.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.984+0000 7f0f71389640 10 monclient(hunting): authenticate will time out at 6476.143456s 2024-03-25T19:37:23.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.995+0000 7f0f6b7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f0f6c1430c0 0x7f0f6c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46688/0 (socket says 172.21.15.134:46688) 2024-03-25T19:37:23.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:23.995+0000 7f0f6b7fe640 1 -- 172.21.15.134:0/1805108770 learned_addr learned my addr 172.21.15.134:0/1805108770 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:24.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.013+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302708357 0 0) 0x7f0f6c142520 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.013+0000 7f0f69ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:24.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.013+0000 7f0f69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:24.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.013+0000 7f0f69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:24.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.013+0000 7f0f69ffb640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:37:24.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.013+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f54003520 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.017+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180179895 0 0) 0x7f0f6c0b2a30 con 0x7f0f6c1430c0 2024-03-25T19:37:24.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.017+0000 7f0f69ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:24.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.017+0000 7f0f69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:24.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.017+0000 7f0f69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:24.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.017+0000 7f0f69ffb640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:37:24.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.017+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f6c142520 con 0x7f0f6c1430c0 2024-03-25T19:37:24.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.018+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730841222 0 0) 0x7f0f6c0b28a0 con 0x7f0f6c13f710 2024-03-25T19:37:24.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.018+0000 7f0f69ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:24.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.018+0000 7f0f69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:24.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.018+0000 7f0f69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:24.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.018+0000 7f0f69ffb640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:37:24.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.018+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f6c0b2a30 con 0x7f0f6c13f710 2024-03-25T19:37:24.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3147356385 0 0) 0x7f0f54003520 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 >> v1:172.21.15.134:6790/0 conn(0x7f0f6c1430c0 legacy=0x7f0f6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 >> v1:172.21.15.159:6789/0 conn(0x7f0f6c13f710 legacy=0x7f0f6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 1 monclient: found mon.noname-a 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c0b44c0 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0f5c001690 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f69ffb640 10 monclient: dump: 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:24.022 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f71389640 5 monclient: authenticate success, global_id 4647 2024-03-25T19:37:24.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.021+0000 7f0f71389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f5c0017f0 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f69ffb640 1 -- 172.21.15.134:0/1805108770 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0f5c001c00 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f69ffb640 10 monclient: dump: 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:24.031 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f71389640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:24.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.031+0000 7f0f71389640 4 set_mon_vals no callback set 2024-03-25T19:37:24.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.032+0000 7f0f71389640 10 monclient: shutdown 2024-03-25T19:37:24.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.032+0000 7f0f71389640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.032+0000 7f0f71389640 1 -- 172.21.15.134:0/1805108770 >> v1:172.21.15.134:6789/0 conn(0x7f0f6c0a98f0 legacy=0x7f0f6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.032+0000 7f0f71389640 1 -- 172.21.15.134:0/1805108770 shutdown_connections 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.032+0000 7f0f71389640 1 -- 172.21.15.134:0/1805108770 >> 172.21.15.134:0/1805108770 conn(0x7f0f6c0b7ea0 msgr2=0x7f0f6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.032+0000 7f0f71389640 1 -- 172.21.15.134:0/1805108770 shutdown_connections 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.032+0000 7f0f71389640 1 -- 172.21.15.134:0/1805108770 wait complete. 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 10 monclient: build_initial_monmap 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 10 monclient: monmap: 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:24.034 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 1 librados: starting msgr at 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 1 librados: starting objecter 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 1 Processor -- start 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 1 -- start start 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 1 librados: setting wanted keys 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 1 librados: calling monclient init 2024-03-25T19:37:24.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.033+0000 7f0f71389640 10 monclient: init 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 10 monclient(hunting): picked mon.noname-a con 0x7f0f6c0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 10 monclient(hunting): picked mon.noname-c con 0x7f0f6c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 10 monclient(hunting): picked mon.noname-b con 0x7f0f6c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c132a30 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c138c80 con 0x7f0f6c1430c0 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c138f60 con 0x7f0f6c13f710 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:24.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.034+0000 7f0f71389640 10 monclient(hunting): authenticate will time out at 6476.193333s 2024-03-25T19:37:24.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.050+0000 7f0f6affd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f0f6c0a98f0 0x7f0f6c137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:60630/0 (socket says 172.21.15.134:60630) 2024-03-25T19:37:24.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.050+0000 7f0f6affd640 1 -- 172.21.15.134:0/2570273591 learned_addr learned my addr 172.21.15.134:0/2570273591 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:24.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.052+0000 7f0f537fe640 10 client.?.objecter ms_handle_connect 0x7f0f6c0a98f0 2024-03-25T19:37:24.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.052+0000 7f0f537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:24.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.052+0000 7f0f537fe640 10 client.?.objecter ms_handle_connect 0x7f0f6c1430c0 2024-03-25T19:37:24.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.052+0000 7f0f537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:24.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.057+0000 7f0f537fe640 10 client.?.objecter ms_handle_connect 0x7f0f6c13f710 2024-03-25T19:37:24.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.057+0000 7f0f537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:24.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.062+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039842169 0 0) 0x7f0f6c138c80 con 0x7f0f6c1430c0 2024-03-25T19:37:24.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.062+0000 7f0f537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:24.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.062+0000 7f0f537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:24.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.062+0000 7f0f537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:24.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.062+0000 7f0f537fe640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:37:24.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.062+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f58004eb0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.065+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216099182 0 0) 0x7f0f6c132a30 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.065+0000 7f0f537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:24.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.065+0000 7f0f537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:24.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.065+0000 7f0f537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:24.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.065+0000 7f0f537fe640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:37:24.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.065+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f6c138c80 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.074+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120453495 0 0) 0x7f0f6c138f60 con 0x7f0f6c13f710 2024-03-25T19:37:24.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.074+0000 7f0f537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:24.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.074+0000 7f0f537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:24.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.074+0000 7f0f537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:24.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.074+0000 7f0f537fe640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:37:24.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.074+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f6c132a30 con 0x7f0f6c13f710 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.074+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3220225700 0 0) 0x7f0f58004eb0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.075+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f6c138f60 con 0x7f0f6c1430c0 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.075+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0f64001bd0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.075+0000 7f0f537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.075+0000 7f0f537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.075+0000 7f0f537fe640 10 monclient(hunting): dump: 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.075 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.076 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:24.076 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:24.076 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:24.076 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:24.076 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:24.076 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:24.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.075+0000 7f0f537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.085+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1842708804 0 0) 0x7f0f6c138c80 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.085+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f58004eb0 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.085+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0f5c001b40 con 0x7f0f6c0a98f0 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.085+0000 7f0f537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.085+0000 7f0f537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.085+0000 7f0f537fe640 10 monclient(hunting): dump: 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.085 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:24.086 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:24.086 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:24.086 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:24.086 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:24.086 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.089+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1788305750 0 0) 0x7f0f6c132a30 con 0x7f0f6c13f710 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.089+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f6c138c80 con 0x7f0f6c13f710 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.089+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0f60001830 con 0x7f0f6c13f710 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.089+0000 7f0f537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.089+0000 7f0f537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.089+0000 7f0f537fe640 10 monclient(hunting): dump: 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:24.089 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:24.090 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:24.090 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:24.090 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:24.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1017355181 0 0) 0x7f0f6c138f60 con 0x7f0f6c1430c0 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 >> v1:172.21.15.134:6790/0 conn(0x7f0f6c13f710 legacy=0x7f0f6c131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 >> v1:172.21.15.134:6789/0 conn(0x7f0f6c0a98f0 legacy=0x7f0f6c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 1 monclient: found mon.b 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c139240 con 0x7f0f6c1430c0 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f71389640 5 monclient: authenticate success, global_id 4612 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f71389640 10 monclient: _renew_subs 2024-03-25T19:37:24.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f71389640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.101+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6c136e50 con 0x7f0f6c1430c0 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 10 monclient: _renew_subs 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6c1389f0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 1 librados: init done 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:24.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:24.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.102+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6c0627b0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.383+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f640020e0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.383+0000 7f0f537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.383+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f0f64004d00 con 0x7f0f6c1430c0 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.383+0000 7f0f537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.383+0000 7f0f537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.383+0000 7f0f537fe640 10 monclient: dump: 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.383 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.383+0000 7f0f697fa640 4 set_mon_vals no callback set 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.384+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f0f6401ad10 con 0x7f0f6c1430c0 2024-03-25T19:37:24.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.384+0000 7f0f6affd640 10 monclient: get_auth_request con 0x7f0f5806e0b0 auth_method 0 2024-03-25T19:37:24.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.384+0000 7f0f537fe640 10 client.4612.objecter ms_handle_connect 0x7f0f5806e0b0 2024-03-25T19:37:24.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f0f64054a80 con 0x7f0f6c1430c0 2024-03-25T19:37:24.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 10 client.4612.objecter ms_dispatch 0x7f0f6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:24.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:24.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:24.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T19:37:24.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f640503c0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:24.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.390+0000 7f0f537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:24.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.641+0000 7f0f71389640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:24.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.641+0000 7f0f71389640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:24.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.641+0000 7f0f71389640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:24.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.641+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f6c13dd60 con 0x7f0f6c1430c0 2024-03-25T19:37:24.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.799+0000 7f0f537fe640 1 -- 172.21.15.134:0/2570273591 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f0f640524c0 con 0x7f0f6c1430c0 2024-03-25T19:37:24.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.799+0000 7f0f537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:24.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.799+0000 7f0f537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:24.800 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:24.800 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:24.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 10 librados: watch_flush enter 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 10 librados: watch_flush exit 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 10 client.4612.objecter successfully canceled tick 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 >> v1:172.21.15.159:6800/1875874887 conn(0x7f0f5806e0b0 legacy=0x7f0f58070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 10 monclient: shutdown 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 >> v1:172.21.15.159:6789/0 conn(0x7f0f6c1430c0 legacy=0x7f0f6c132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 shutdown_connections 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 >> 172.21.15.134:0/2570273591 conn(0x7f0f6c0b7ea0 msgr2=0x7f0f6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 shutdown_connections 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 1 -- 172.21.15.134:0/2570273591 wait complete. 2024-03-25T19:37:24.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:24.802+0000 7f0f71389640 1 librados: shutdown 2024-03-25T19:37:24.815 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:26.151 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:30.823 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.207 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:30.879 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.243 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:30.890 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.249 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:30.921 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.262 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:30.909 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.264 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:30.936 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.280 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:30.952 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.284 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:30.931 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:26.306 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:30.978 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:31.621 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.740+0000 7f2eff54d640 10 monclient: get_monmap_and_config 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.740+0000 7f2eff54d640 10 monclient: build_initial_monmap 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.740+0000 7f2eff54d640 10 monclient: monmap: 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:31.741384+0000 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:31.741384+0000 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:31.740 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:31.741 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:31.741 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:31.741 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.741+0000 7f2eff54d640 1 Processor -- start 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.741+0000 7f2eff54d640 1 -- start start 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.741+0000 7f2eff54d640 10 monclient: init 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ef80b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef80b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef814be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:31.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef80af940 con 0x7f2ef80b1730 2024-03-25T19:37:31.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef80b0a70 con 0x7f2ef814be90 2024-03-25T19:37:31.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef80b0d50 con 0x7f2ef80b4d60 2024-03-25T19:37:31.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:31.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.742+0000 7f2eff54d640 10 monclient(hunting): authenticate will time out at 6483.900853s 2024-03-25T19:37:31.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.751+0000 7f2efdaea640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f2ef814be90 0x7f2ef814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:43334/0 (socket says 172.21.15.134:43334) 2024-03-25T19:37:31.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.751+0000 7f2efdaea640 1 -- 172.21.15.134:0/2132479439 learned_addr learned my addr 172.21.15.134:0/2132479439 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:31.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.762+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371602295 0 0) 0x7f2ef80b0a70 con 0x7f2ef814be90 2024-03-25T19:37:31.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.762+0000 7f2eeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:31.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.762+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:31.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.762+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:31.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.762+0000 7f2eeffff640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:37:31.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.762+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee4003520 con 0x7f2ef814be90 2024-03-25T19:37:31.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.767+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346182876 0 0) 0x7f2ef80b0d50 con 0x7f2ef80b4d60 2024-03-25T19:37:31.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.767+0000 7f2eeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:31.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.767+0000 7f2eeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:31.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.767+0000 7f2eeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:31.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.767+0000 7f2eeffff640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:37:31.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.767+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef80b0a70 con 0x7f2ef80b4d60 2024-03-25T19:37:31.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727775146 0 0) 0x7f2ee4003520 con 0x7f2ef814be90 2024-03-25T19:37:31.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 >> v1:172.21.15.134:6790/0 conn(0x7f2ef80b4d60 legacy=0x7f2ef813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:31.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 >> v1:172.21.15.134:6789/0 conn(0x7f2ef80b1730 legacy=0x7f2ef80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:31.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:31.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 1 monclient: found mon.noname-c 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef80b1030 con 0x7f2ef814be90 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 10 monclient: _finish_auth 0 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2ef0001a10 con 0x7f2ef814be90 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eeffff640 10 monclient: dump: 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:31.777 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:31.778 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:31.778 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:31.778 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:31.778 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:31.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eff54d640 5 monclient: authenticate success, global_id 4615 2024-03-25T19:37:31.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.776+0000 7f2eff54d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ef0001b70 con 0x7f2ef814be90 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eeffff640 1 -- 172.21.15.134:0/2132479439 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2ef0001f80 con 0x7f2ef814be90 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eeffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eeffff640 10 monclient: dump: 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.792 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eff54d640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:31.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.792+0000 7f2eff54d640 4 set_mon_vals no callback set 2024-03-25T19:37:31.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.793+0000 7f2eff54d640 10 monclient: shutdown 2024-03-25T19:37:31.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.793+0000 7f2eff54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:31.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.793+0000 7f2eff54d640 1 -- 172.21.15.134:0/2132479439 >> v1:172.21.15.159:6789/0 conn(0x7f2ef814be90 legacy=0x7f2ef814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:31.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.793+0000 7f2eff54d640 1 -- 172.21.15.134:0/2132479439 shutdown_connections 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.793+0000 7f2eff54d640 1 -- 172.21.15.134:0/2132479439 >> 172.21.15.134:0/2132479439 conn(0x7f2ef80617a0 msgr2=0x7f2ef8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.793+0000 7f2eff54d640 1 -- 172.21.15.134:0/2132479439 shutdown_connections 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.793+0000 7f2eff54d640 1 -- 172.21.15.134:0/2132479439 wait complete. 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 10 monclient: build_initial_monmap 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 10 monclient: monmap: 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:31.796 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 1 librados: starting msgr at 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 1 librados: starting objecter 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 1 Processor -- start 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 1 -- start start 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 1 librados: setting wanted keys 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 1 librados: calling monclient init 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.794+0000 7f2eff54d640 10 monclient: init 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ef80b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef80b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef814be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:31.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef8140380 con 0x7f2ef80b4d60 2024-03-25T19:37:31.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef8140660 con 0x7f2ef80b1730 2024-03-25T19:37:31.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef8141e30 con 0x7f2ef814be90 2024-03-25T19:37:31.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:31.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.795+0000 7f2eff54d640 10 monclient(hunting): authenticate will time out at 6483.954196s 2024-03-25T19:37:31.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.803+0000 7f2efcae8640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2ef80b4d60 0x7f2ef8142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34778/0 (socket says 172.21.15.134:34778) 2024-03-25T19:37:31.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.803+0000 7f2efcae8640 1 -- 172.21.15.134:0/1500980358 learned_addr learned my addr 172.21.15.134:0/1500980358 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:31.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.806+0000 7f2eedffb640 10 client.?.objecter ms_handle_connect 0x7f2ef80b4d60 2024-03-25T19:37:31.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.806+0000 7f2eedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:31.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.811+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872966419 0 0) 0x7f2ef8140380 con 0x7f2ef80b4d60 2024-03-25T19:37:31.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.811+0000 7f2eedffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:31.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.811+0000 7f2eedffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:31.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.811+0000 7f2eedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:31.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.811+0000 7f2eedffb640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:37:31.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.811+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2edc004eb0 con 0x7f2ef80b4d60 2024-03-25T19:37:31.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.818+0000 7f2eedffb640 10 client.?.objecter ms_handle_connect 0x7f2ef80b1730 2024-03-25T19:37:31.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.818+0000 7f2eedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:31.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.821+0000 7f2eedffb640 10 client.?.objecter ms_handle_connect 0x7f2ef814be90 2024-03-25T19:37:31.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.821+0000 7f2eedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073527046 0 0) 0x7f2ef8141e30 con 0x7f2ef814be90 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef8140380 con 0x7f2ef814be90 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143497754 0 0) 0x7f2ef8140660 con 0x7f2ef80b1730 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:37:31.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.831+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef8141e30 con 0x7f2ef80b1730 2024-03-25T19:37:31.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.834+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724677435 0 0) 0x7f2edc004eb0 con 0x7f2ef80b4d60 2024-03-25T19:37:31.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.834+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ef8140660 con 0x7f2ef80b4d60 2024-03-25T19:37:31.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.834+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2ee8001a30 con 0x7f2ef80b4d60 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.834+0000 7f2eedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.834+0000 7f2eedffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.834+0000 7f2eedffb640 10 monclient(hunting): dump: 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:31.835 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:31.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.834+0000 7f2eedffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.845+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2180268854 0 0) 0x7f2ef8141e30 con 0x7f2ef80b1730 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.845+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2edc004eb0 con 0x7f2ef80b1730 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.845+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2ef4001930 con 0x7f2ef80b1730 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.845+0000 7f2eedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.845+0000 7f2eedffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.845+0000 7f2eedffb640 10 monclient(hunting): dump: 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:31.845 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:31.846 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:31.846 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:31.846 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:31.846 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:31.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 377095451 0 0) 0x7f2ef8140660 con 0x7f2ef80b4d60 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 >> v1:172.21.15.134:6790/0 conn(0x7f2ef814be90 legacy=0x7f2ef81e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 >> v1:172.21.15.159:6789/0 conn(0x7f2ef80b1730 legacy=0x7f2ef8138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 1 monclient: found mon.a 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8142110 con 0x7f2ef80b4d60 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eff54d640 5 monclient: authenticate success, global_id 4656 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eff54d640 10 monclient: _renew_subs 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eff54d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:31.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.848+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef8140e60 con 0x7f2ef80b4d60 2024-03-25T19:37:31.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.849+0000 7f2eff54d640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:31.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.849+0000 7f2eff54d640 10 monclient: _renew_subs 2024-03-25T19:37:31.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.849+0000 7f2eff54d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:31.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.849+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef8142c40 con 0x7f2ef80b4d60 2024-03-25T19:37:31.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.849+0000 7f2eff54d640 1 librados: init done 2024-03-25T19:37:31.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.850+0000 7f2eff54d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:31.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.850+0000 7f2eff54d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:31.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.850+0000 7f2eff54d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:31.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:31.850+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef80627b0 con 0x7f2ef80b4d60 2024-03-25T19:37:32.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.069+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ee8001c70 con 0x7f2ef80b4d60 2024-03-25T19:37:32.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.069+0000 7f2eedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:32.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.069+0000 7f2eef7fe640 4 set_mon_vals no callback set 2024-03-25T19:37:32.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.149+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f2ee8003a60 con 0x7f2ef80b4d60 2024-03-25T19:37:32.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.149+0000 7f2eedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:32.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.149+0000 7f2eedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.149+0000 7f2eedffb640 10 monclient: dump: 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:32.150 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:32.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.150+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f2ee8003c40 con 0x7f2ef80b4d60 2024-03-25T19:37:32.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.151+0000 7f2efd2e9640 10 monclient: get_auth_request con 0x7f2edc06df20 auth_method 0 2024-03-25T19:37:32.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.151+0000 7f2eedffb640 10 client.4656.objecter ms_handle_connect 0x7f2edc06df20 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f2ee8054950 con 0x7f2ef80b4d60 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 10 client.4656.objecter ms_dispatch 0x7f2ef8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 3 client.4656.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 3 client.4656.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ee8050460 con 0x7f2ef80b4d60 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.157+0000 7f2eedffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:32.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.412+0000 7f2eff54d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:32.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.412+0000 7f2eff54d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:32.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.412+0000 7f2eff54d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:37:32.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.412+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ef8140810 con 0x7f2ef80b4d60 2024-03-25T19:37:32.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.442+0000 7f2eedffb640 1 -- 172.21.15.134:0/1500980358 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f2ef8140810 con 0x7f2ef80b4d60 2024-03-25T19:37:32.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.442+0000 7f2eedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:32.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.442+0000 7f2eedffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:32.442 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:32.442 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.444+0000 7f2eff54d640 10 librados: watch_flush enter 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 10 librados: watch_flush exit 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 10 client.4656.objecter successfully canceled tick 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 >> v1:172.21.15.159:6800/1875874887 conn(0x7f2edc06df20 legacy=0x7f2edc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 10 monclient: shutdown 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 >> v1:172.21.15.134:6789/0 conn(0x7f2ef80b4d60 legacy=0x7f2ef8142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 shutdown_connections 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 >> 172.21.15.134:0/1500980358 conn(0x7f2ef80617a0 msgr2=0x7f2ef80b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:32.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 shutdown_connections 2024-03-25T19:37:32.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 1 -- 172.21.15.134:0/1500980358 wait complete. 2024-03-25T19:37:32.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:32.445+0000 7f2eff54d640 1 librados: shutdown 2024-03-25T19:37:32.456 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:39.262 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:39.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.378+0000 7efc260fb640 10 monclient: get_monmap_and_config 2024-03-25T19:37:39.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.378+0000 7efc260fb640 10 monclient: build_initial_monmap 2024-03-25T19:37:39.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.378+0000 7efc260fb640 10 monclient: monmap: 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:39.379696+0000 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:39.379696+0000 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:39.379 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.379+0000 7efc260fb640 1 Processor -- start 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.379+0000 7efc260fb640 1 -- start start 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.379+0000 7efc260fb640 10 monclient: init 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 10 monclient(hunting): picked mon.noname-a con 0x7efc200a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 10 monclient(hunting): picked mon.noname-b con 0x7efc2013f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 10 monclient(hunting): picked mon.noname-c con 0x7efc201430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc20142520 con 0x7efc200a98f0 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc200b28a0 con 0x7efc201430c0 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc200b2a30 con 0x7efc2013f710 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.380+0000 7efc260fb640 10 monclient(hunting): authenticate will time out at 6491.539158s 2024-03-25T19:37:39.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.386+0000 7efc1effd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7efc2013f710 0x7efc201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:51896/0 (socket says 172.21.15.134:51896) 2024-03-25T19:37:39.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.386+0000 7efc1effd640 1 -- 172.21.15.134:0/951943656 learned_addr learned my addr 172.21.15.134:0/951943656 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:39.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.394+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744552676 0 0) 0x7efc200b28a0 con 0x7efc201430c0 2024-03-25T19:37:39.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.394+0000 7efc1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:39.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.394+0000 7efc1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:39.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.394+0000 7efc1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:39.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.394+0000 7efc1e7fc640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:37:39.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.394+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc08003520 con 0x7efc201430c0 2024-03-25T19:37:39.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.403+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376070137 0 0) 0x7efc200b2a30 con 0x7efc2013f710 2024-03-25T19:37:39.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.403+0000 7efc1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:39.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.403+0000 7efc1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:39.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.403+0000 7efc1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:39.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.403+0000 7efc1e7fc640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:37:39.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.403+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc200b28a0 con 0x7efc2013f710 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 300499028 0 0) 0x7efc08003520 con 0x7efc201430c0 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 >> v1:172.21.15.134:6790/0 conn(0x7efc2013f710 legacy=0x7efc201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 >> v1:172.21.15.134:6789/0 conn(0x7efc200a98f0 legacy=0x7efc200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc200b44c0 con 0x7efc201430c0 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:39.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efc18001a10 con 0x7efc201430c0 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc1e7fc640 10 monclient: dump: 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc260fb640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:37:39.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.405+0000 7efc260fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc18001b70 con 0x7efc201430c0 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc1e7fc640 1 -- 172.21.15.134:0/951943656 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efc18001f80 con 0x7efc201430c0 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc1e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc1e7fc640 10 monclient: dump: 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.424 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc260fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:39.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.424+0000 7efc260fb640 4 set_mon_vals no callback set 2024-03-25T19:37:39.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.425+0000 7efc260fb640 10 monclient: shutdown 2024-03-25T19:37:39.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.425+0000 7efc260fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:39.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.425+0000 7efc260fb640 1 -- 172.21.15.134:0/951943656 >> v1:172.21.15.159:6789/0 conn(0x7efc201430c0 legacy=0x7efc2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:39.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.425+0000 7efc260fb640 1 -- 172.21.15.134:0/951943656 shutdown_connections 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.425+0000 7efc260fb640 1 -- 172.21.15.134:0/951943656 >> 172.21.15.134:0/951943656 conn(0x7efc200b7ea0 msgr2=0x7efc200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.425+0000 7efc260fb640 1 -- 172.21.15.134:0/951943656 shutdown_connections 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.425+0000 7efc260fb640 1 -- 172.21.15.134:0/951943656 wait complete. 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.426+0000 7efc260fb640 10 monclient: build_initial_monmap 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.426+0000 7efc260fb640 10 monclient: monmap: 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:39.427 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.426+0000 7efc260fb640 1 librados: starting msgr at 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.426+0000 7efc260fb640 1 librados: starting objecter 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.426+0000 7efc260fb640 1 Processor -- start 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 1 -- start start 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 1 librados: setting wanted keys 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 1 librados: calling monclient init 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient: init 2024-03-25T19:37:39.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient(hunting): picked mon.noname-c con 0x7efc200a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient(hunting): picked mon.noname-a con 0x7efc2013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient(hunting): picked mon.noname-b con 0x7efc201430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc201354c0 con 0x7efc2013f710 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc201319b0 con 0x7efc201430c0 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc20135070 con 0x7efc200a98f0 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:39.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.427+0000 7efc260fb640 10 monclient(hunting): authenticate will time out at 6491.586522s 2024-03-25T19:37:39.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.437+0000 7efc1effd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7efc2013f710 0x7efc20130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39436/0 (socket says 172.21.15.134:39436) 2024-03-25T19:37:39.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.437+0000 7efc1effd640 1 -- 172.21.15.134:0/745916969 learned_addr learned my addr 172.21.15.134:0/745916969 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:39.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.439+0000 7efc07fff640 10 client.?.objecter ms_handle_connect 0x7efc2013f710 2024-03-25T19:37:39.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.439+0000 7efc07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:39.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.441+0000 7efc07fff640 10 client.?.objecter ms_handle_connect 0x7efc201430c0 2024-03-25T19:37:39.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.441+0000 7efc07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:39.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.442+0000 7efc07fff640 10 client.?.objecter ms_handle_connect 0x7efc200a98f0 2024-03-25T19:37:39.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.442+0000 7efc07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:39.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.449+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838532440 0 0) 0x7efc20135070 con 0x7efc200a98f0 2024-03-25T19:37:39.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.449+0000 7efc07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.449+0000 7efc07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.449+0000 7efc07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.449+0000 7efc07fff640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.449+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc0c004eb0 con 0x7efc200a98f0 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.450+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843744719 0 0) 0x7efc201319b0 con 0x7efc201430c0 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.450+0000 7efc07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.450+0000 7efc07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.450+0000 7efc07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.450+0000 7efc07fff640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:37:39.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.450+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc20135070 con 0x7efc201430c0 2024-03-25T19:37:39.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.455+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602845892 0 0) 0x7efc201354c0 con 0x7efc2013f710 2024-03-25T19:37:39.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.455+0000 7efc07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:39.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.455+0000 7efc07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:39.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.455+0000 7efc07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:39.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.455+0000 7efc07fff640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:37:39.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.455+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc201319b0 con 0x7efc2013f710 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.476+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1896063965 0 0) 0x7efc20135070 con 0x7efc201430c0 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.476+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc201354c0 con 0x7efc201430c0 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.476+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efc18001e80 con 0x7efc201430c0 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.476+0000 7efc07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.476+0000 7efc07fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.476+0000 7efc07fff640 10 monclient(hunting): dump: 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:39.476 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:39.477 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:39.477 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.476+0000 7efc07fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:39.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1473413961 0 0) 0x7efc201319b0 con 0x7efc2013f710 2024-03-25T19:37:39.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc20135070 con 0x7efc2013f710 2024-03-25T19:37:39.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efc140019b0 con 0x7efc2013f710 2024-03-25T19:37:39.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:39.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:39.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 10 monclient(hunting): dump: 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4585080 0 0) 0x7efc0c004eb0 con 0x7efc200a98f0 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc201319b0 con 0x7efc200a98f0 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efc10001930 con 0x7efc200a98f0 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:39.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.480+0000 7efc07fff640 10 monclient(hunting): dump: 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:39.482 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 519293257 0 0) 0x7efc201354c0 con 0x7efc201430c0 2024-03-25T19:37:39.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 >> v1:172.21.15.134:6790/0 conn(0x7efc200a98f0 legacy=0x7efc201d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 >> v1:172.21.15.134:6789/0 conn(0x7efc2013f710 legacy=0x7efc20130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 1 monclient: found mon.b 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc20135200 con 0x7efc201430c0 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 10 monclient: _finish_auth 0 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 5 monclient: authenticate success, global_id 4624 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 10 monclient: _renew_subs 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc200b7010 con 0x7efc201430c0 2024-03-25T19:37:39.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 10 monclient: _renew_subs 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2013b890 con 0x7efc201430c0 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.486+0000 7efc260fb640 1 librados: init done 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.487+0000 7efc260fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.487+0000 7efc260fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.487+0000 7efc260fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:39.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.487+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc20137160 con 0x7efc201430c0 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.771+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc18001fe0 con 0x7efc201430c0 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.771+0000 7efc07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.771+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efc18003dc0 con 0x7efc201430c0 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.771+0000 7efc07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.771+0000 7efc07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.771+0000 7efc07fff640 10 monclient: dump: 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:39.772 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.771+0000 7efc1dffb640 4 set_mon_vals no callback set 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.772+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7efc18019d90 con 0x7efc201430c0 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.773+0000 7efc1f7fe640 10 monclient: get_auth_request con 0x7efc0c06e0b0 auth_method 0 2024-03-25T19:37:39.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.773+0000 7efc07fff640 10 client.4624.objecter ms_handle_connect 0x7efc0c06e0b0 2024-03-25T19:37:39.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7efc180538e0 con 0x7efc201430c0 2024-03-25T19:37:39.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 10 client.4624.objecter ms_dispatch 0x7efc20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:39.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:39.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:39.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T19:37:39.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc1804f3f0 con 0x7efc201430c0 2024-03-25T19:37:39.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:39.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:39.778+0000 7efc07fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:40.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.023+0000 7efc260fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:40.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.023+0000 7efc260fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:40.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.023+0000 7efc260fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:37:40.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.023+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc20135670 con 0x7efc201430c0 2024-03-25T19:37:40.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.057+0000 7efc07fff640 1 -- 172.21.15.134:0/745916969 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7efc20135670 con 0x7efc201430c0 2024-03-25T19:37:40.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.057+0000 7efc07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:40.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.057+0000 7efc07fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:40.057 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:40.057 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:40.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 10 librados: watch_flush enter 2024-03-25T19:37:40.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 10 librados: watch_flush exit 2024-03-25T19:37:40.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 10 client.4624.objecter successfully canceled tick 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 >> v1:172.21.15.159:6800/1875874887 conn(0x7efc0c06e0b0 legacy=0x7efc0c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 10 monclient: shutdown 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 >> v1:172.21.15.159:6789/0 conn(0x7efc201430c0 legacy=0x7efc201312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 shutdown_connections 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 >> 172.21.15.134:0/745916969 conn(0x7efc200b7ea0 msgr2=0x7efc2013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 shutdown_connections 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 1 -- 172.21.15.134:0/745916969 wait complete. 2024-03-25T19:37:40.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:40.060+0000 7efc260fb640 1 librados: shutdown 2024-03-25T19:37:40.072 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:46.220 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:50.893 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.270 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:50.943 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.314 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:50.986 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.333 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:51.006 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.336 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:50.983 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.341 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:50.988 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.346 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:51.018 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.361 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:51.008 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.368 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:51.041 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:46.880 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:46.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:46.999+0000 7f814c36d640 10 monclient: get_monmap_and_config 2024-03-25T19:37:46.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:46.999+0000 7f814c36d640 10 monclient: build_initial_monmap 2024-03-25T19:37:46.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:46.999+0000 7f814c36d640 10 monclient: monmap: 2024-03-25T19:37:46.999 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:46.999 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:46.999 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:47.000745+0000 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:47.000745+0000 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.000+0000 7f814c36d640 1 Processor -- start 2024-03-25T19:37:47.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.000+0000 7f814c36d640 1 -- start start 2024-03-25T19:37:47.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.000+0000 7f814c36d640 10 monclient: init 2024-03-25T19:37:47.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 10 monclient(hunting): picked mon.noname-a con 0x7f81440a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 10 monclient(hunting): picked mon.noname-c con 0x7f814413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 10 monclient(hunting): picked mon.noname-b con 0x7f81441430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8144142520 con 0x7f81440a98f0 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81440b28a0 con 0x7f814413f710 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81440b2a30 con 0x7f81441430c0 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:47.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.001+0000 7f814c36d640 10 monclient(hunting): authenticate will time out at 6499.160378s 2024-03-25T19:37:47.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.012+0000 7f814a109640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f81440a98f0 0x7f81440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:52404/0 (socket says 172.21.15.134:52404) 2024-03-25T19:37:47.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.012+0000 7f814a109640 1 -- 172.21.15.134:0/955507966 learned_addr learned my addr 172.21.15.134:0/955507966 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:47.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.028+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764196010 0 0) 0x7f81440b28a0 con 0x7f814413f710 2024-03-25T19:37:47.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.028+0000 7f8149107640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.028+0000 7f8149107640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.028+0000 7f8149107640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.028+0000 7f8149107640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:37:47.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.028+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8130003520 con 0x7f814413f710 2024-03-25T19:37:47.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.035+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242903439 0 0) 0x7f8144142520 con 0x7f81440a98f0 2024-03-25T19:37:47.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.035+0000 7f8149107640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.035+0000 7f8149107640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.035+0000 7f8149107640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.035+0000 7f8149107640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:37:47.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.035+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81440b28a0 con 0x7f81440a98f0 2024-03-25T19:37:47.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750341652 0 0) 0x7f8130003520 con 0x7f814413f710 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 >> v1:172.21.15.134:6790/0 conn(0x7f81441430c0 legacy=0x7f814414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 >> v1:172.21.15.134:6789/0 conn(0x7f81440a98f0 legacy=0x7f81440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 10 monclient: _finish_hunting 0 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 1 monclient: found mon.noname-c 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81440b44c0 con 0x7f814413f710 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 10 monclient: _finish_auth 0 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 10 monclient: _check_auth_tickets 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f814c36d640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8140001690 con 0x7f814413f710 2024-03-25T19:37:47.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f814c36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f8149107640 10 monclient: dump: 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.037+0000 7f814c36d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:47.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81400017f0 con 0x7f814413f710 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f8149107640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f8149107640 1 -- 172.21.15.134:0/955507966 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8140001c00 con 0x7f814413f710 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f8149107640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f8149107640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f8149107640 10 monclient: dump: 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:47.053 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f814c36d640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.051+0000 7f814c36d640 4 set_mon_vals no callback set 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.052+0000 7f814c36d640 10 monclient: shutdown 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.052+0000 7f814c36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.052+0000 7f814c36d640 1 -- 172.21.15.134:0/955507966 >> v1:172.21.15.159:6789/0 conn(0x7f814413f710 legacy=0x7f81441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.053+0000 7f814c36d640 1 -- 172.21.15.134:0/955507966 shutdown_connections 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.053+0000 7f814c36d640 1 -- 172.21.15.134:0/955507966 >> 172.21.15.134:0/955507966 conn(0x7f81440b7ea0 msgr2=0x7f81440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.053+0000 7f814c36d640 1 -- 172.21.15.134:0/955507966 shutdown_connections 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.053+0000 7f814c36d640 1 -- 172.21.15.134:0/955507966 wait complete. 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 10 monclient: build_initial_monmap 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 10 monclient: monmap: 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:47.055 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 1 librados: starting msgr at 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 1 librados: starting objecter 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 1 Processor -- start 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 1 -- start start 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 1 librados: setting wanted keys 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 1 librados: calling monclient init 2024-03-25T19:37:47.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.054+0000 7f814c36d640 10 monclient: init 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 10 monclient(hunting): picked mon.noname-c con 0x7f81440a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 10 monclient(hunting): picked mon.noname-a con 0x7f814413f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 10 monclient(hunting): picked mon.noname-b con 0x7f81441430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81440abdc0 con 0x7f814413f710 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81441d7730 con 0x7f81441430c0 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8144138f10 con 0x7f81440a98f0 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:47.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.055+0000 7f814c36d640 10 monclient(hunting): authenticate will time out at 6499.213886s 2024-03-25T19:37:47.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.073+0000 7f814a109640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f81440a98f0 0x7f8144138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:40822/0 (socket says 172.21.15.134:40822) 2024-03-25T19:37:47.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.073+0000 7f814a109640 1 -- 172.21.15.134:0/4290973436 learned_addr learned my addr 172.21.15.134:0/4290973436 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:47.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.075+0000 7f813affd640 10 client.?.objecter ms_handle_connect 0x7f81440a98f0 2024-03-25T19:37:47.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.075+0000 7f813affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:47.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.077+0000 7f813affd640 10 client.?.objecter ms_handle_connect 0x7f81441430c0 2024-03-25T19:37:47.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.077+0000 7f813affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:47.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.083+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509024133 0 0) 0x7f8144138f10 con 0x7f81440a98f0 2024-03-25T19:37:47.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.083+0000 7f813affd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.083+0000 7f813affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.083+0000 7f813affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.083+0000 7f813affd640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:37:47.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.083+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8128004eb0 con 0x7f81440a98f0 2024-03-25T19:37:47.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.086+0000 7f813affd640 10 client.?.objecter ms_handle_connect 0x7f814413f710 2024-03-25T19:37:47.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.086+0000 7f813affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:47.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.088+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710409297 0 0) 0x7f81441d7730 con 0x7f81441430c0 2024-03-25T19:37:47.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.088+0000 7f813affd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.088+0000 7f813affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.088+0000 7f813affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.088+0000 7f813affd640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:37:47.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.088+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8144138f10 con 0x7f81441430c0 2024-03-25T19:37:47.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.096+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235547245 0 0) 0x7f81440abdc0 con 0x7f814413f710 2024-03-25T19:37:47.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.096+0000 7f813affd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.096+0000 7f813affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.096+0000 7f813affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.096+0000 7f813affd640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:37:47.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.096+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81441d7730 con 0x7f814413f710 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.103+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4146192516 0 0) 0x7f8128004eb0 con 0x7f81440a98f0 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.103+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81440abdc0 con 0x7f81440a98f0 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.103+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f81340019b0 con 0x7f81440a98f0 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.103+0000 7f813affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.103+0000 7f813affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.103+0000 7f813affd640 10 monclient(hunting): dump: 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:47.103 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.103+0000 7f813affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.109+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81985801 0 0) 0x7f81441d7730 con 0x7f814413f710 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.109+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8128004eb0 con 0x7f814413f710 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.109+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8140001b00 con 0x7f814413f710 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.109+0000 7f813affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.109+0000 7f813affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.109+0000 7f813affd640 10 monclient(hunting): dump: 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:47.109 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:47.110 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.122+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242440626 0 0) 0x7f8144138f10 con 0x7f81441430c0 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.122+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81441d7730 con 0x7f81441430c0 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.122+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f813c001b00 con 0x7f81441430c0 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.122+0000 7f813affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.122+0000 7f813affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.122+0000 7f813affd640 10 monclient(hunting): dump: 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.122 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.123 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.123 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:47.123 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:47.123 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:47.123 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:47.123 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1025054825 0 0) 0x7f81440abdc0 con 0x7f81440a98f0 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 >> v1:172.21.15.159:6789/0 conn(0x7f81441430c0 legacy=0x7f81441d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 >> v1:172.21.15.134:6789/0 conn(0x7f814413f710 legacy=0x7f81441d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 1 monclient: found mon.c 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81441390a0 con 0x7f81440a98f0 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 10 monclient: _finish_auth 0 2024-03-25T19:37:47.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f813affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f814c36d640 5 monclient: authenticate success, global_id 4637 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f814c36d640 10 monclient: _renew_subs 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f814c36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.133+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81441d8320 con 0x7f81440a98f0 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.134+0000 7f814c36d640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.134+0000 7f814c36d640 10 monclient: _renew_subs 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.134+0000 7f814c36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.134+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81441d43c0 con 0x7f81440a98f0 2024-03-25T19:37:47.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.134+0000 7f814c36d640 1 librados: init done 2024-03-25T19:37:47.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.135+0000 7f814c36d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:47.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.135+0000 7f814c36d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:47.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.135+0000 7f814c36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:47.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.135+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81440627b0 con 0x7f81440a98f0 2024-03-25T19:37:47.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.366+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81340047e0 con 0x7f81440a98f0 2024-03-25T19:37:47.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.366+0000 7f813affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:47.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.366+0000 7f8148906640 4 set_mon_vals no callback set 2024-03-25T19:37:47.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.439+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8134004af0 con 0x7f81440a98f0 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.439+0000 7f813affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.439+0000 7f813affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.439+0000 7f813affd640 10 monclient: dump: 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:47.439 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:47.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.439+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f813401aac0 con 0x7f81440a98f0 2024-03-25T19:37:47.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.440+0000 7f8149908640 10 monclient: get_auth_request con 0x7f812806e0f0 auth_method 0 2024-03-25T19:37:47.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.440+0000 7f813affd640 10 client.4637.objecter ms_handle_connect 0x7f812806e0f0 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f8134054910 con 0x7f81440a98f0 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 10 client.4637.objecter ms_dispatch 0x7f8144000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8134050250 con 0x7f81440a98f0 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.446+0000 7f813affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:47.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.677+0000 7f814c36d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:47.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.677+0000 7f814c36d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:47.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.677+0000 7f814c36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:47.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.677+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8144069f40 con 0x7f81440a98f0 2024-03-25T19:37:47.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.707+0000 7f813affd640 1 -- 172.21.15.134:0/4290973436 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f8134052350 con 0x7f81440a98f0 2024-03-25T19:37:47.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.707+0000 7f813affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:47.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.707+0000 7f813affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:47.708 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:47.708 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:47.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 10 librados: watch_flush enter 2024-03-25T19:37:47.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 10 librados: watch_flush exit 2024-03-25T19:37:47.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 10 client.4637.objecter successfully canceled tick 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 >> v1:172.21.15.159:6800/1875874887 conn(0x7f812806e0f0 legacy=0x7f81280705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 10 monclient: shutdown 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 >> v1:172.21.15.134:6790/0 conn(0x7f81440a98f0 legacy=0x7f8144138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 shutdown_connections 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 >> 172.21.15.134:0/4290973436 conn(0x7f81440b7ea0 msgr2=0x7f8144074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 shutdown_connections 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 1 -- 172.21.15.134:0/4290973436 wait complete. 2024-03-25T19:37:47.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:47.710+0000 7f814c36d640 1 librados: shutdown 2024-03-25T19:37:47.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:54.530 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:54.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.631+0000 7f1b94d7c640 10 monclient: get_monmap_and_config 2024-03-25T19:37:54.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.631+0000 7f1b94d7c640 10 monclient: build_initial_monmap 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.631+0000 7f1b94d7c640 10 monclient: monmap: 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:37:54.632106+0000 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:37:54.632106+0000 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:37:54.631 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.631+0000 7f1b94d7c640 1 Processor -- start 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.631+0000 7f1b94d7c640 1 -- start start 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.631+0000 7f1b94d7c640 10 monclient: init 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 10 monclient(hunting): picked mon.noname-a con 0x7f1b900b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 10 monclient(hunting): picked mon.noname-c con 0x7f1b900b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 10 monclient(hunting): picked mon.noname-b con 0x7f1b9014be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900af940 con 0x7f1b900b4d60 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900b0a70 con 0x7f1b900b1730 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900b0d50 con 0x7f1b9014be90 2024-03-25T19:37:54.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:54.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.632+0000 7f1b94d7c640 10 monclient(hunting): authenticate will time out at 6506.791514s 2024-03-25T19:37:54.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.648+0000 7f1b8ed9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f1b9014be90 0x7f1b9014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50070/0 (socket says 172.21.15.134:50070) 2024-03-25T19:37:54.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.648+0000 7f1b8ed9d640 1 -- 172.21.15.134:0/865666969 learned_addr learned my addr 172.21.15.134:0/865666969 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:54.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.663+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086086033 0 0) 0x7f1b900b0a70 con 0x7f1b900b1730 2024-03-25T19:37:54.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.664+0000 7f1b8d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:54.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.664+0000 7f1b8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:54.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.664+0000 7f1b8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:54.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.664+0000 7f1b8d59a640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:37:54.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.664+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b78003520 con 0x7f1b900b1730 2024-03-25T19:37:54.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.666+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966120952 0 0) 0x7f1b900b0d50 con 0x7f1b9014be90 2024-03-25T19:37:54.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.666+0000 7f1b8d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:54.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.666+0000 7f1b8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:54.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.666+0000 7f1b8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:54.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.666+0000 7f1b8d59a640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:37:54.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.666+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b900b0a70 con 0x7f1b9014be90 2024-03-25T19:37:54.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196258580 0 0) 0x7f1b900b0a70 con 0x7f1b9014be90 2024-03-25T19:37:54.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 >> v1:172.21.15.159:6789/0 conn(0x7f1b900b1730 legacy=0x7f1b900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:54.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 >> v1:172.21.15.134:6789/0 conn(0x7f1b900b4d60 legacy=0x7f1b9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:54.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:37:54.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 1 monclient: found mon.noname-b 2024-03-25T19:37:54.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b900b1030 con 0x7f1b9014be90 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 10 monclient: _finish_auth 0 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1b880019f0 con 0x7f1b9014be90 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b8d59a640 10 monclient: dump: 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:54.676 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b94d7c640 5 monclient: authenticate success, global_id 4640 2024-03-25T19:37:54.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.675+0000 7f1b94d7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:54.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b88001b50 con 0x7f1b9014be90 2024-03-25T19:37:54.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b8d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:54.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b8d59a640 1 -- 172.21.15.134:0/865666969 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1b88001f20 con 0x7f1b9014be90 2024-03-25T19:37:54.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b8d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b8d59a640 10 monclient: dump: 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:54.703 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:54.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b94d7c640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:54.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.702+0000 7f1b94d7c640 4 set_mon_vals no callback set 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.703+0000 7f1b94d7c640 10 monclient: shutdown 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.703+0000 7f1b94d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.703+0000 7f1b94d7c640 1 -- 172.21.15.134:0/865666969 >> v1:172.21.15.134:6790/0 conn(0x7f1b9014be90 legacy=0x7f1b9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.703+0000 7f1b94d7c640 1 -- 172.21.15.134:0/865666969 shutdown_connections 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.703+0000 7f1b94d7c640 1 -- 172.21.15.134:0/865666969 >> 172.21.15.134:0/865666969 conn(0x7f1b900617a0 msgr2=0x7f1b90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.703+0000 7f1b94d7c640 1 -- 172.21.15.134:0/865666969 shutdown_connections 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.703+0000 7f1b94d7c640 1 -- 172.21.15.134:0/865666969 wait complete. 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.704+0000 7f1b94d7c640 10 monclient: build_initial_monmap 2024-03-25T19:37:54.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.704+0000 7f1b94d7c640 10 monclient: monmap: 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.705+0000 7f1b94d7c640 1 librados: starting msgr at 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.705+0000 7f1b94d7c640 1 librados: starting objecter 2024-03-25T19:37:54.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.705+0000 7f1b94d7c640 1 Processor -- start 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.705+0000 7f1b94d7c640 1 -- start start 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.705+0000 7f1b94d7c640 1 librados: setting wanted keys 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.705+0000 7f1b94d7c640 1 librados: calling monclient init 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.705+0000 7f1b94d7c640 10 monclient: init 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 10 monclient(hunting): picked mon.noname-c con 0x7f1b900b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 10 monclient(hunting): picked mon.noname-a con 0x7f1b900b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 10 monclient(hunting): picked mon.noname-b con 0x7f1b9014be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b901327e0 con 0x7f1b900b4d60 2024-03-25T19:37:54.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b90143320 con 0x7f1b9014be90 2024-03-25T19:37:54.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b90138080 con 0x7f1b900b1730 2024-03-25T19:37:54.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:54.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.706+0000 7f1b94d7c640 10 monclient(hunting): authenticate will time out at 6506.865103s 2024-03-25T19:37:54.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.713+0000 7f1b8e59c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f1b900b1730 0x7f1b901420c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50120/0 (socket says 172.21.15.134:50120) 2024-03-25T19:37:54.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.713+0000 7f1b8e59c640 1 -- 172.21.15.134:0/3580116909 learned_addr learned my addr 172.21.15.134:0/3580116909 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:37:54.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.715+0000 7f1b7effd640 10 client.?.objecter ms_handle_connect 0x7f1b900b1730 2024-03-25T19:37:54.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.715+0000 7f1b7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:54.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.723+0000 7f1b7effd640 10 client.?.objecter ms_handle_connect 0x7f1b9014be90 2024-03-25T19:37:54.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.723+0000 7f1b7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:54.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.725+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213570529 0 0) 0x7f1b90138080 con 0x7f1b900b1730 2024-03-25T19:37:54.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.725+0000 7f1b7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:54.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.725+0000 7f1b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:54.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.725+0000 7f1b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:54.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.725+0000 7f1b7effd640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:37:54.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.725+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b70004eb0 con 0x7f1b900b1730 2024-03-25T19:37:54.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.726+0000 7f1b7effd640 10 client.?.objecter ms_handle_connect 0x7f1b900b4d60 2024-03-25T19:37:54.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.726+0000 7f1b7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:54.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.740+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405339694 0 0) 0x7f1b90143320 con 0x7f1b9014be90 2024-03-25T19:37:54.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.740+0000 7f1b7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:54.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.740+0000 7f1b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:54.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.740+0000 7f1b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:54.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.740+0000 7f1b7effd640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:37:54.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.740+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b90138080 con 0x7f1b9014be90 2024-03-25T19:37:54.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.743+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849068803 0 0) 0x7f1b901327e0 con 0x7f1b900b4d60 2024-03-25T19:37:54.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.743+0000 7f1b7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:54.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.743+0000 7f1b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:54.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.743+0000 7f1b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:54.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.743+0000 7f1b7effd640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:37:54.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.743+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b90143320 con 0x7f1b900b4d60 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.750+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153257123 0 0) 0x7f1b70004eb0 con 0x7f1b900b1730 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.750+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b901327e0 con 0x7f1b900b1730 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.750+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1b80001a30 con 0x7f1b900b1730 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.750+0000 7f1b7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.750+0000 7f1b7effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.750+0000 7f1b7effd640 10 monclient(hunting): dump: 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:54.750 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:54.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.750+0000 7f1b7effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.763+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917734640 0 0) 0x7f1b90143320 con 0x7f1b900b4d60 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.763+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b70004eb0 con 0x7f1b900b4d60 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.763+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1b840017b0 con 0x7f1b900b4d60 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.763+0000 7f1b7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.763+0000 7f1b7effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.763+0000 7f1b7effd640 10 monclient(hunting): dump: 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.765 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2888668305 0 0) 0x7f1b901327e0 con 0x7f1b900b1730 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 >> v1:172.21.15.159:6789/0 conn(0x7f1b9014be90 legacy=0x7f1b901320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 >> v1:172.21.15.134:6789/0 conn(0x7f1b900b4d60 legacy=0x7f1b901319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:54.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 1 monclient: found mon.c 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b90137c60 con 0x7f1b900b1730 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 10 monclient: _finish_auth 0 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 5 monclient: authenticate success, global_id 4643 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 10 monclient: _renew_subs 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b901427d0 con 0x7f1b900b1730 2024-03-25T19:37:54.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 10 monclient: _renew_subs 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b90140560 con 0x7f1b900b1730 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.764+0000 7f1b94d7c640 1 librados: init done 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.765+0000 7f1b94d7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.765+0000 7f1b94d7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.765+0000 7f1b94d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:54.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:54.765+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b90141ac0 con 0x7f1b900b1730 2024-03-25T19:37:55.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b80001c70 con 0x7f1b900b1730 2024-03-25T19:37:55.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:55.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1b80003a60 con 0x7f1b900b1730 2024-03-25T19:37:55.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:55.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:55.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b7effd640 10 monclient: dump: 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b8cd99640 4 set_mon_vals no callback set 2024-03-25T19:37:55.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.080+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f1b80003c40 con 0x7f1b900b1730 2024-03-25T19:37:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.081+0000 7f1b8dd9b640 10 monclient: get_auth_request con 0x7f1b7006df70 auth_method 0 2024-03-25T19:37:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.081+0000 7f1b7effd640 10 client.4643.objecter ms_handle_connect 0x7f1b7006df70 2024-03-25T19:37:55.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f1b800548b0 con 0x7f1b900b1730 2024-03-25T19:37:55.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 10 client.4643.objecter ms_dispatch 0x7f1b90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:55.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:55.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:55.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T19:37:55.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b800503c0 con 0x7f1b900b1730 2024-03-25T19:37:55.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:55.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.086+0000 7f1b7effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:55.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.338+0000 7f1b94d7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:55.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.338+0000 7f1b94d7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:55.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.338+0000 7f1b94d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:37:55.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.338+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b900627b0 con 0x7f1b900b1730 2024-03-25T19:37:55.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.365+0000 7f1b7effd640 1 -- 172.21.15.134:0/3580116909 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f1b800522f0 con 0x7f1b900b1730 2024-03-25T19:37:55.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.365+0000 7f1b7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:55.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.365+0000 7f1b7effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:55.365 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:37:55.365 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 10 librados: watch_flush enter 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 10 librados: watch_flush exit 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 10 client.4643.objecter successfully canceled tick 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 >> v1:172.21.15.159:6800/1875874887 conn(0x7f1b7006df70 legacy=0x7f1b70070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 10 monclient: shutdown 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 >> v1:172.21.15.134:6790/0 conn(0x7f1b900b1730 legacy=0x7f1b901420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:55.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 shutdown_connections 2024-03-25T19:37:55.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.367+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 >> 172.21.15.134:0/3580116909 conn(0x7f1b900617a0 msgr2=0x7f1b9013c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:55.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.368+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 shutdown_connections 2024-03-25T19:37:55.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.368+0000 7f1b94d7c640 1 -- 172.21.15.134:0/3580116909 wait complete. 2024-03-25T19:37:55.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:37:55.368+0000 7f1b94d7c640 1 librados: shutdown 2024-03-25T19:37:55.379 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:02.185 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.299+0000 7f09d28c0640 10 monclient: get_monmap_and_config 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.299+0000 7f09d28c0640 10 monclient: build_initial_monmap 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.299+0000 7f09d28c0640 10 monclient: monmap: 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:02.300530+0000 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:02.300530+0000 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:02.299 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:02.300 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:02.300 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:02.300 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.300+0000 7f09d28c0640 1 Processor -- start 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.300+0000 7f09d28c0640 1 -- start start 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.300+0000 7f09d28c0640 10 monclient: init 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0af940 con 0x7f09cc14be90 2024-03-25T19:38:02.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0b0a70 con 0x7f09cc0b1730 2024-03-25T19:38:02.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0b0d50 con 0x7f09cc0b4d60 2024-03-25T19:38:02.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:02.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.301+0000 7f09d28c0640 10 monclient(hunting): authenticate will time out at 6514.459782s 2024-03-25T19:38:02.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.315+0000 7f09cb7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f09cc0b1730 0x7f09cc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:42308/0 (socket says 172.21.15.134:42308) 2024-03-25T19:38:02.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.315+0000 7f09cb7fe640 1 -- 172.21.15.134:0/1756676226 learned_addr learned my addr 172.21.15.134:0/1756676226 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:02.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.321+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620253604 0 0) 0x7f09cc0b0d50 con 0x7f09cc0b4d60 2024-03-25T19:38:02.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.321+0000 7f09caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:02.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.321+0000 7f09caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:02.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.321+0000 7f09caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:02.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.321+0000 7f09caffd640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:38:02.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.321+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b4003520 con 0x7f09cc0b4d60 2024-03-25T19:38:02.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.329+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739925737 0 0) 0x7f09cc0b0a70 con 0x7f09cc0b1730 2024-03-25T19:38:02.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.329+0000 7f09caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:02.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.329+0000 7f09caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:02.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.329+0000 7f09caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:02.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.329+0000 7f09caffd640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:38:02.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.329+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc0b0d50 con 0x7f09cc0b1730 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2412440938 0 0) 0x7f09b4003520 con 0x7f09cc0b4d60 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 >> v1:172.21.15.159:6789/0 conn(0x7f09cc0b1730 legacy=0x7f09cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 >> v1:172.21.15.134:6789/0 conn(0x7f09cc14be90 legacy=0x7f09cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 1 monclient: found mon.noname-b 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc0b1030 con 0x7f09cc0b4d60 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.332+0000 7f09caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.333+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f09bc001690 con 0x7f09cc0b4d60 2024-03-25T19:38:02.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.333+0000 7f09caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.333+0000 7f09caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.333+0000 7f09caffd640 10 monclient: dump: 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.333+0000 7f09d28c0640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:38:02.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.333+0000 7f09d28c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:02.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09bc0017f0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:02.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09caffd640 1 -- 172.21.15.134:0/1756676226 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f09bc001c00 con 0x7f09cc0b4d60 2024-03-25T19:38:02.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:02.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09caffd640 10 monclient: dump: 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09d28c0640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:02.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.341+0000 7f09d28c0640 4 set_mon_vals no callback set 2024-03-25T19:38:02.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.342+0000 7f09d28c0640 10 monclient: shutdown 2024-03-25T19:38:02.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.342+0000 7f09d28c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:02.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.342+0000 7f09d28c0640 1 -- 172.21.15.134:0/1756676226 >> v1:172.21.15.134:6790/0 conn(0x7f09cc0b4d60 legacy=0x7f09cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.342+0000 7f09d28c0640 1 -- 172.21.15.134:0/1756676226 shutdown_connections 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.342+0000 7f09d28c0640 1 -- 172.21.15.134:0/1756676226 >> 172.21.15.134:0/1756676226 conn(0x7f09cc0617a0 msgr2=0x7f09cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.342+0000 7f09d28c0640 1 -- 172.21.15.134:0/1756676226 shutdown_connections 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.342+0000 7f09d28c0640 1 -- 172.21.15.134:0/1756676226 wait complete. 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.343+0000 7f09d28c0640 10 monclient: build_initial_monmap 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.343+0000 7f09d28c0640 10 monclient: monmap: 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:02.344 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:02.345 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:02.345 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.344+0000 7f09d28c0640 1 librados: starting msgr at 2024-03-25T19:38:02.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.344+0000 7f09d28c0640 1 librados: starting objecter 2024-03-25T19:38:02.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.344+0000 7f09d28c0640 1 Processor -- start 2024-03-25T19:38:02.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.344+0000 7f09d28c0640 1 -- start start 2024-03-25T19:38:02.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.344+0000 7f09d28c0640 1 librados: setting wanted keys 2024-03-25T19:38:02.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.344+0000 7f09d28c0640 1 librados: calling monclient init 2024-03-25T19:38:02.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.344+0000 7f09d28c0640 10 monclient: init 2024-03-25T19:38:02.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:02.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:02.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:02.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:02.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:02.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc1d7ba0 con 0x7f09cc14be90 2024-03-25T19:38:02.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc1d8740 con 0x7f09cc0b1730 2024-03-25T19:38:02.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc1d7830 con 0x7f09cc0b4d60 2024-03-25T19:38:02.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:02.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.345+0000 7f09d28c0640 10 monclient(hunting): authenticate will time out at 6514.504116s 2024-03-25T19:38:02.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.353+0000 7f09cb7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f09cc0b4d60 0x7f09cc135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:57120/0 (socket says 172.21.15.134:57120) 2024-03-25T19:38:02.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.353+0000 7f09cb7fe640 1 -- 172.21.15.134:0/2492392061 learned_addr learned my addr 172.21.15.134:0/2492392061 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:02.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.358+0000 7f09c8ff9640 10 client.?.objecter ms_handle_connect 0x7f09cc0b4d60 2024-03-25T19:38:02.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.358+0000 7f09c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:02.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.369+0000 7f09c8ff9640 10 client.?.objecter ms_handle_connect 0x7f09cc14be90 2024-03-25T19:38:02.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.369+0000 7f09c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:02.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.372+0000 7f09c8ff9640 10 client.?.objecter ms_handle_connect 0x7f09cc0b1730 2024-03-25T19:38:02.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.372+0000 7f09c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:02.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.374+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891433933 0 0) 0x7f09cc1d7830 con 0x7f09cc0b4d60 2024-03-25T19:38:02.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.374+0000 7f09c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:02.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.374+0000 7f09c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:02.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.374+0000 7f09c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:02.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.374+0000 7f09c8ff9640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:38:02.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.374+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b8004eb0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.383+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546113290 0 0) 0x7f09cc1d8740 con 0x7f09cc0b1730 2024-03-25T19:38:02.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.383+0000 7f09c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:02.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.383+0000 7f09c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:02.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.383+0000 7f09c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:02.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.383+0000 7f09c8ff9640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:38:02.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.383+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc1d7830 con 0x7f09cc0b1730 2024-03-25T19:38:02.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.385+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701055299 0 0) 0x7f09cc1d7ba0 con 0x7f09cc14be90 2024-03-25T19:38:02.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.385+0000 7f09c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:02.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.385+0000 7f09c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:02.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.385+0000 7f09c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:02.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.385+0000 7f09c8ff9640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:38:02.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.385+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc1d8740 con 0x7f09cc14be90 2024-03-25T19:38:02.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892152149 0 0) 0x7f09b8004eb0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09cc1d7ba0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f09c00019b0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2832680055 0 0) 0x7f09cc1d7830 con 0x7f09cc0b1730 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09b8004eb0 con 0x7f09cc0b1730 2024-03-25T19:38:02.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f09bc001b00 con 0x7f09cc0b1730 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.395+0000 7f09c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:02.397 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.396+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1636589024 0 0) 0x7f09cc1d8740 con 0x7f09cc14be90 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.396+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09cc1d7830 con 0x7f09cc14be90 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.396+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f09c4001a90 con 0x7f09cc14be90 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.396+0000 7f09c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.396+0000 7f09c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.396+0000 7f09c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:02.398 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:02.399 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:02.399 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:02.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:02.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619791443 0 0) 0x7f09cc1d7ba0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 >> v1:172.21.15.159:6789/0 conn(0x7f09cc0b1730 legacy=0x7f09cc136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:02.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 >> v1:172.21.15.134:6789/0 conn(0x7f09cc14be90 legacy=0x7f09cc1d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:02.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:02.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 1 monclient: found mon.c 2024-03-25T19:38:02.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc143900 con 0x7f09cc0b4d60 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 10 monclient: _renew_subs 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc143bb0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 10 monclient: _renew_subs 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc130060 con 0x7f09cc0b4d60 2024-03-25T19:38:02.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.411+0000 7f09d28c0640 1 librados: init done 2024-03-25T19:38:02.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.412+0000 7f09d28c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:02.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.412+0000 7f09d28c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:02.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.412+0000 7f09d28c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:02.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.412+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc0627b0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.717+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09c00047e0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.717+0000 7f09c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.717+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f09c0004af0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.717+0000 7f09c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.717+0000 7f09c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.717+0000 7f09c8ff9640 10 monclient: dump: 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:02.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.717+0000 7f09ca7fc640 4 set_mon_vals no callback set 2024-03-25T19:38:02.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.718+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f09c001ab80 con 0x7f09cc0b4d60 2024-03-25T19:38:02.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.719+0000 7f09cbfff640 10 monclient: get_auth_request con 0x7f09b806e0b0 auth_method 0 2024-03-25T19:38:02.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.719+0000 7f09c8ff9640 10 client.4649.objecter ms_handle_connect 0x7f09b806e0b0 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f09c00548b0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 10 client.4649.objecter ms_dispatch 0x7f09cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09c00501f0 con 0x7f09cc0b4d60 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.723+0000 7f09c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:02.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.945+0000 7f09d28c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:02.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.945+0000 7f09d28c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:02.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.945+0000 7f09d28c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:02.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:02.945+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09cc1d7d50 con 0x7f09cc0b4d60 2024-03-25T19:38:03.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.010+0000 7f09c8ff9640 1 -- 172.21.15.134:0/2492392061 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f09cc1d7d50 con 0x7f09cc0b4d60 2024-03-25T19:38:03.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.010+0000 7f09c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:03.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.010+0000 7f09c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:03.010 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:03.010 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.012+0000 7f09d28c0640 10 librados: watch_flush enter 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.012+0000 7f09d28c0640 10 librados: watch_flush exit 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.012+0000 7f09d28c0640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.012+0000 7f09d28c0640 10 client.4649.objecter successfully canceled tick 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 >> v1:172.21.15.159:6800/1875874887 conn(0x7f09b806e0b0 legacy=0x7f09b8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 10 monclient: shutdown 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 >> v1:172.21.15.134:6790/0 conn(0x7f09cc0b4d60 legacy=0x7f09cc135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 shutdown_connections 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 >> 172.21.15.134:0/2492392061 conn(0x7f09cc0617a0 msgr2=0x7f09cc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 shutdown_connections 2024-03-25T19:38:03.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 1 -- 172.21.15.134:0/2492392061 wait complete. 2024-03-25T19:38:03.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:03.013+0000 7f09d28c0640 1 librados: shutdown 2024-03-25T19:38:03.026 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:06.421 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:11.093 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.433 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:11.105 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.462 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:11.135 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.486 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:11.159 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.489 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:11.136 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.492 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:11.164 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.492 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:11.139 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.519 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:11.191 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:06.520 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:11.167 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:09.834 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:09.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.952+0000 7fc60f863640 10 monclient: get_monmap_and_config 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.952+0000 7fc60f863640 10 monclient: build_initial_monmap 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.952+0000 7fc60f863640 10 monclient: monmap: 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:09.953753+0000 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:09.953753+0000 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:09.953 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.953+0000 7fc60f863640 1 Processor -- start 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 1 -- start start 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient: init 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient(hunting): picked mon.noname-a con 0x7fc6080a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient(hunting): picked mon.noname-b con 0x7fc60813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient(hunting): picked mon.noname-c con 0x7fc6081430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc608142520 con 0x7fc6080a98f0 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6080b28a0 con 0x7fc6081430c0 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6080b2a30 con 0x7fc60813f710 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:09.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.954+0000 7fc60f863640 10 monclient(hunting): authenticate will time out at 6522.113619s 2024-03-25T19:38:09.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.965+0000 7fc60cdfe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc60813f710 0x7fc6081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33488/0 (socket says 172.21.15.134:33488) 2024-03-25T19:38:09.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.965+0000 7fc60cdfe640 1 -- 172.21.15.134:0/2132780342 learned_addr learned my addr 172.21.15.134:0/2132780342 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:09.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.974+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971033669 0 0) 0x7fc6080b2a30 con 0x7fc60813f710 2024-03-25T19:38:09.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.974+0000 7fc5fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:09.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.974+0000 7fc5fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:09.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.974+0000 7fc5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:09.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.974+0000 7fc5fffff640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:38:09.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.974+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f4003520 con 0x7fc60813f710 2024-03-25T19:38:09.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.991+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282528031 0 0) 0x7fc608142520 con 0x7fc6080a98f0 2024-03-25T19:38:09.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.991+0000 7fc5fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:09.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.991+0000 7fc5fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:09.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.991+0000 7fc5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:09.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.991+0000 7fc5fffff640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.991+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6080b2a30 con 0x7fc6080a98f0 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3662015609 0 0) 0x7fc5f4003520 con 0x7fc60813f710 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 >> v1:172.21.15.159:6789/0 conn(0x7fc6081430c0 legacy=0x7fc60814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 >> v1:172.21.15.134:6789/0 conn(0x7fc6080a98f0 legacy=0x7fc6080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 1 monclient: found mon.noname-b 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6080b44c0 con 0x7fc60813f710 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 10 monclient: _finish_auth 0 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc5f8001690 con 0x7fc60813f710 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc60f863640 5 monclient: authenticate success, global_id 4652 2024-03-25T19:38:09.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc60f863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc5fffff640 10 monclient: dump: 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:09.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.992+0000 7fc60f863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5f80017f0 con 0x7fc60813f710 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc5fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc5fffff640 1 -- 172.21.15.134:0/2132780342 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc5f8001c00 con 0x7fc60813f710 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc5fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc5fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc5fffff640 10 monclient: dump: 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:09.996 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc60f863640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:09.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.996+0000 7fc60f863640 4 set_mon_vals no callback set 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.997+0000 7fc60f863640 10 monclient: shutdown 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.997+0000 7fc60f863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.997+0000 7fc60f863640 1 -- 172.21.15.134:0/2132780342 >> v1:172.21.15.134:6790/0 conn(0x7fc60813f710 legacy=0x7fc6081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.997+0000 7fc60f863640 1 -- 172.21.15.134:0/2132780342 shutdown_connections 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.997+0000 7fc60f863640 1 -- 172.21.15.134:0/2132780342 >> 172.21.15.134:0/2132780342 conn(0x7fc6080b7ea0 msgr2=0x7fc6080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.997+0000 7fc60f863640 1 -- 172.21.15.134:0/2132780342 shutdown_connections 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.997+0000 7fc60f863640 1 -- 172.21.15.134:0/2132780342 wait complete. 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.998+0000 7fc60f863640 10 monclient: build_initial_monmap 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.998+0000 7fc60f863640 10 monclient: monmap: 2024-03-25T19:38:09.999 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.999+0000 7fc60f863640 1 librados: starting msgr at 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.999+0000 7fc60f863640 1 librados: starting objecter 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.999+0000 7fc60f863640 1 Processor -- start 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.999+0000 7fc60f863640 1 -- start start 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.999+0000 7fc60f863640 1 librados: setting wanted keys 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.999+0000 7fc60f863640 1 librados: calling monclient init 2024-03-25T19:38:10.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:09.999+0000 7fc60f863640 10 monclient: init 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 10 monclient(hunting): picked mon.noname-c con 0x7fc6080a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 10 monclient(hunting): picked mon.noname-a con 0x7fc60813f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 10 monclient(hunting): picked mon.noname-b con 0x7fc6081430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6080ab8b0 con 0x7fc60813f710 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6080abb90 con 0x7fc6081430c0 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc608137870 con 0x7fc6080a98f0 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:10.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.000+0000 7fc60f863640 10 monclient(hunting): authenticate will time out at 6522.159027s 2024-03-25T19:38:10.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.012+0000 7fc60d5ff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc6080a98f0 0x7fc6081350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33522/0 (socket says 172.21.15.134:33522) 2024-03-25T19:38:10.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.012+0000 7fc60d5ff640 1 -- 172.21.15.134:0/2622780172 learned_addr learned my addr 172.21.15.134:0/2622780172 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:10.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.013+0000 7fc5fdffb640 10 client.?.objecter ms_handle_connect 0x7fc6080a98f0 2024-03-25T19:38:10.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.013+0000 7fc5fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:10.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.016+0000 7fc5fdffb640 10 client.?.objecter ms_handle_connect 0x7fc6081430c0 2024-03-25T19:38:10.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.016+0000 7fc5fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:10.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.020+0000 7fc5fdffb640 10 client.?.objecter ms_handle_connect 0x7fc60813f710 2024-03-25T19:38:10.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.020+0000 7fc5fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:10.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.022+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653679604 0 0) 0x7fc6080abb90 con 0x7fc6081430c0 2024-03-25T19:38:10.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.022+0000 7fc5fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.022+0000 7fc5fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.022+0000 7fc5fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.022+0000 7fc5fdffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:38:10.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.022+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ec004eb0 con 0x7fc6081430c0 2024-03-25T19:38:10.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.023+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543845851 0 0) 0x7fc608137870 con 0x7fc6080a98f0 2024-03-25T19:38:10.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.023+0000 7fc5fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.023+0000 7fc5fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.023+0000 7fc5fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.023+0000 7fc5fdffb640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:38:10.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.023+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6080abb90 con 0x7fc6080a98f0 2024-03-25T19:38:10.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.031+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033016107 0 0) 0x7fc6080ab8b0 con 0x7fc60813f710 2024-03-25T19:38:10.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.031+0000 7fc5fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.031+0000 7fc5fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.031+0000 7fc5fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.031+0000 7fc5fdffb640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:38:10.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.031+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc608137870 con 0x7fc60813f710 2024-03-25T19:38:10.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.036+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3325069341 0 0) 0x7fc6080abb90 con 0x7fc6080a98f0 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.036+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6080ab8b0 con 0x7fc6080a98f0 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.036+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc604001a70 con 0x7fc6080a98f0 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.036+0000 7fc5fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.036+0000 7fc5fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.036+0000 7fc5fdffb640 10 monclient(hunting): dump: 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:10.036 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:10.037 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:10.037 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:10.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.036+0000 7fc5fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.045+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1603691249 0 0) 0x7fc5ec004eb0 con 0x7fc6081430c0 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.045+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6080abb90 con 0x7fc6081430c0 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.045+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc600001b20 con 0x7fc6081430c0 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.045+0000 7fc5fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.045+0000 7fc5fdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.045+0000 7fc5fdffb640 10 monclient(hunting): dump: 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:10.045 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.046 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:10.046 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:10.046 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:10.046 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:10.046 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:10.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2925138038 0 0) 0x7fc6080ab8b0 con 0x7fc6080a98f0 2024-03-25T19:38:10.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 >> v1:172.21.15.159:6789/0 conn(0x7fc6081430c0 legacy=0x7fc6081e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 >> v1:172.21.15.134:6789/0 conn(0x7fc60813f710 legacy=0x7fc6081df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:10.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 1 monclient: found mon.c 2024-03-25T19:38:10.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6081389e0 con 0x7fc6080a98f0 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc5fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 5 monclient: authenticate success, global_id 4655 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 10 monclient: _renew_subs 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc608136d90 con 0x7fc6080a98f0 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 10 monclient: _renew_subs 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:10.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6081384c0 con 0x7fc6080a98f0 2024-03-25T19:38:10.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.047+0000 7fc60f863640 1 librados: init done 2024-03-25T19:38:10.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.048+0000 7fc60f863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:10.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.048+0000 7fc60f863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:10.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.048+0000 7fc60f863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:10.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.048+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc608136f80 con 0x7fc6080a98f0 2024-03-25T19:38:10.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.358+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc604001c50 con 0x7fc6080a98f0 2024-03-25T19:38:10.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.358+0000 7fc5fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:10.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.358+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc604003a60 con 0x7fc6080a98f0 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.358+0000 7fc5fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.358+0000 7fc5fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.358+0000 7fc5fdffb640 10 monclient: dump: 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:10.359 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:10.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.358+0000 7fc5ff7fe640 4 set_mon_vals no callback set 2024-03-25T19:38:10.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.359+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fc604003c80 con 0x7fc6080a98f0 2024-03-25T19:38:10.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.360+0000 7fc60cdfe640 10 monclient: get_auth_request con 0x7fc5ec06df70 auth_method 0 2024-03-25T19:38:10.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.360+0000 7fc5fdffb640 10 client.4655.objecter ms_handle_connect 0x7fc5ec06df70 2024-03-25T19:38:10.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.365+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7fc604054890 con 0x7fc6080a98f0 2024-03-25T19:38:10.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.365+0000 7fc5fdffb640 10 client.4655.objecter ms_dispatch 0x7fc608000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:10.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.365+0000 7fc5fdffb640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:10.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.365+0000 7fc5fdffb640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:10.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.365+0000 7fc5fdffb640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-25T19:38:10.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.366+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6040503a0 con 0x7fc6080a98f0 2024-03-25T19:38:10.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.366+0000 7fc5fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:10.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.366+0000 7fc5fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:10.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.605+0000 7fc60f863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:10.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.605+0000 7fc60f863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:10.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.605+0000 7fc60f863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:10.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.605+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc608138b70 con 0x7fc6080a98f0 2024-03-25T19:38:10.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.649+0000 7fc5fdffb640 1 -- 172.21.15.134:0/2622780172 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7fc6040522d0 con 0x7fc6080a98f0 2024-03-25T19:38:10.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.649+0000 7fc5fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:10.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.649+0000 7fc5fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:10.649 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:10.649 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:10.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 10 librados: watch_flush enter 2024-03-25T19:38:10.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 10 librados: watch_flush exit 2024-03-25T19:38:10.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 10 client.4655.objecter successfully canceled tick 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 >> v1:172.21.15.159:6800/1875874887 conn(0x7fc5ec06df70 legacy=0x7fc5ec070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 10 monclient: shutdown 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 >> v1:172.21.15.134:6790/0 conn(0x7fc6080a98f0 legacy=0x7fc6081350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 shutdown_connections 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 >> 172.21.15.134:0/2622780172 conn(0x7fc6080b7ea0 msgr2=0x7fc608074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 shutdown_connections 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 1 -- 172.21.15.134:0/2622780172 wait complete. 2024-03-25T19:38:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:10.652+0000 7fc60f863640 1 librados: shutdown 2024-03-25T19:38:10.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:17.471 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:17.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.586+0000 7f1548990640 10 monclient: get_monmap_and_config 2024-03-25T19:38:17.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.586+0000 7f1548990640 10 monclient: build_initial_monmap 2024-03-25T19:38:17.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.586+0000 7f1548990640 10 monclient: monmap: 2024-03-25T19:38:17.586 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:17.586 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:17.586 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:17.587802+0000 2024-03-25T19:38:17.587 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:17.587802+0000 2024-03-25T19:38:17.587 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:17.587 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:17.587 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:17.587 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:17.587 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:17.587 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:17.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.587+0000 7f1548990640 1 Processor -- start 2024-03-25T19:38:17.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.587+0000 7f1548990640 1 -- start start 2024-03-25T19:38:17.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.587+0000 7f1548990640 10 monclient: init 2024-03-25T19:38:17.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:17.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 10 monclient(hunting): picked mon.noname-b con 0x7f15440a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 10 monclient(hunting): picked mon.noname-c con 0x7f154413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 10 monclient(hunting): picked mon.noname-a con 0x7f15441430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1544142520 con 0x7f15441430c0 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15440b28a0 con 0x7f154413f710 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15440b2a30 con 0x7f15440a98f0 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:17.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.588+0000 7f1548990640 10 monclient(hunting): authenticate will time out at 6529.747204s 2024-03-25T19:38:17.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.601+0000 7f153bfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f154413f710 0x7f15441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:60392/0 (socket says 172.21.15.134:60392) 2024-03-25T19:38:17.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.601+0000 7f153bfff640 1 -- 172.21.15.134:0/817754795 learned_addr learned my addr 172.21.15.134:0/817754795 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:17.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.610+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889352411 0 0) 0x7f15440b28a0 con 0x7f154413f710 2024-03-25T19:38:17.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.610+0000 7f1541d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:17.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.610+0000 7f1541d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:17.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.610+0000 7f1541d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:17.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.610+0000 7f1541d9b640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:38:17.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.610+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1530003520 con 0x7f154413f710 2024-03-25T19:38:17.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.620+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536428348 0 0) 0x7f15440b2a30 con 0x7f15440a98f0 2024-03-25T19:38:17.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:17.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:17.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:17.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:38:17.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15440b28a0 con 0x7f15440a98f0 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233489423 0 0) 0x7f1530003520 con 0x7f154413f710 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 >> v1:172.21.15.134:6790/0 conn(0x7f15440a98f0 legacy=0x7f15440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 >> v1:172.21.15.134:6789/0 conn(0x7f15441430c0 legacy=0x7f154414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 1 monclient: found mon.noname-c 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15440b44c0 con 0x7f154413f710 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient: _finish_auth 0 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f152c001690 con 0x7f154413f710 2024-03-25T19:38:17.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.621+0000 7f1541d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.622+0000 7f1541d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.622+0000 7f1541d9b640 10 monclient: dump: 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.622+0000 7f1548990640 5 monclient: authenticate success, global_id 4651 2024-03-25T19:38:17.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.622+0000 7f1548990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:17.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f152c0017f0 con 0x7f154413f710 2024-03-25T19:38:17.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1541d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:17.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1541d9b640 1 -- 172.21.15.134:0/817754795 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f152c001c00 con 0x7f154413f710 2024-03-25T19:38:17.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1541d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:17.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1541d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:17.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1541d9b640 10 monclient: dump: 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1548990640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:17.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.630+0000 7f1548990640 4 set_mon_vals no callback set 2024-03-25T19:38:17.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.631+0000 7f1548990640 10 monclient: shutdown 2024-03-25T19:38:17.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.631+0000 7f1548990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.631+0000 7f1548990640 1 -- 172.21.15.134:0/817754795 >> v1:172.21.15.159:6789/0 conn(0x7f154413f710 legacy=0x7f15441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.631+0000 7f1548990640 1 -- 172.21.15.134:0/817754795 shutdown_connections 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.631+0000 7f1548990640 1 -- 172.21.15.134:0/817754795 >> 172.21.15.134:0/817754795 conn(0x7f15440b7ea0 msgr2=0x7f15440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.631+0000 7f1548990640 1 -- 172.21.15.134:0/817754795 shutdown_connections 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.631+0000 7f1548990640 1 -- 172.21.15.134:0/817754795 wait complete. 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.632+0000 7f1548990640 10 monclient: build_initial_monmap 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.632+0000 7f1548990640 10 monclient: monmap: 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:17.634 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.632+0000 7f1548990640 1 librados: starting msgr at 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.632+0000 7f1548990640 1 librados: starting objecter 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.632+0000 7f1548990640 1 Processor -- start 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 1 -- start start 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 1 librados: setting wanted keys 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 1 librados: calling monclient init 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient: init 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient(hunting): picked mon.noname-c con 0x7f15440a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient(hunting): picked mon.noname-b con 0x7f154413f710 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:17.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient(hunting): picked mon.noname-a con 0x7f15441430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:17.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15440abde0 con 0x7f15441430c0 2024-03-25T19:38:17.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15441d7730 con 0x7f154413f710 2024-03-25T19:38:17.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15441d8360 con 0x7f15440a98f0 2024-03-25T19:38:17.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:17.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.633+0000 7f1548990640 10 monclient(hunting): authenticate will time out at 6529.792461s 2024-03-25T19:38:17.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.642+0000 7f1542d9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f15441430c0 0x7f1544138630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:57384/0 (socket says 172.21.15.134:57384) 2024-03-25T19:38:17.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.642+0000 7f1542d9d640 1 -- 172.21.15.134:0/1030658008 learned_addr learned my addr 172.21.15.134:0/1030658008 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:17.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.645+0000 7f153affd640 10 client.?.objecter ms_handle_connect 0x7f15441430c0 2024-03-25T19:38:17.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.645+0000 7f153affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:17.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.646+0000 7f153affd640 10 client.?.objecter ms_handle_connect 0x7f15440a98f0 2024-03-25T19:38:17.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.646+0000 7f153affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:17.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.648+0000 7f153affd640 10 client.?.objecter ms_handle_connect 0x7f154413f710 2024-03-25T19:38:17.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.648+0000 7f153affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:17.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.658+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559151867 0 0) 0x7f15441d7730 con 0x7f154413f710 2024-03-25T19:38:17.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.658+0000 7f153affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:17.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.658+0000 7f153affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:17.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.658+0000 7f153affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:17.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.658+0000 7f153affd640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:38:17.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.658+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1524004eb0 con 0x7f154413f710 2024-03-25T19:38:17.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.661+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637041131 0 0) 0x7f15440abde0 con 0x7f15441430c0 2024-03-25T19:38:17.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.661+0000 7f153affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:17.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.661+0000 7f153affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:17.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.661+0000 7f153affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:17.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.661+0000 7f153affd640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:38:17.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.661+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15441d7730 con 0x7f15441430c0 2024-03-25T19:38:17.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.662+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916433137 0 0) 0x7f15441d8360 con 0x7f15440a98f0 2024-03-25T19:38:17.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.662+0000 7f153affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:17.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.662+0000 7f153affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:17.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.662+0000 7f153affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:17.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.662+0000 7f153affd640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:38:17.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.662+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15440abde0 con 0x7f15440a98f0 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.678+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 202539548 0 0) 0x7f15441d7730 con 0x7f15441430c0 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.678+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15441d8360 con 0x7f15441430c0 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.678+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f153c001bd0 con 0x7f15441430c0 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.678+0000 7f153affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.678+0000 7f153affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.678+0000 7f153affd640 10 monclient(hunting): dump: 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:17.678 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:17.679 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:17.679 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:17.679 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:17.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.678+0000 7f153affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.689+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475420837 0 0) 0x7f15440abde0 con 0x7f15440a98f0 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.689+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15441d7730 con 0x7f15440a98f0 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.689+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f15340019b0 con 0x7f15440a98f0 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.689+0000 7f153affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.689+0000 7f153affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.689+0000 7f153affd640 10 monclient(hunting): dump: 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:17.689 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:17.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.691+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2258190865 0 0) 0x7f1524004eb0 con 0x7f154413f710 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.691+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15440abde0 con 0x7f154413f710 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.691+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f152c001b00 con 0x7f154413f710 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.691+0000 7f153affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.691+0000 7f153affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.691+0000 7f153affd640 10 monclient(hunting): dump: 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:17.691 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:17.692 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:17.692 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:17.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3428512323 0 0) 0x7f15441d8360 con 0x7f15441430c0 2024-03-25T19:38:17.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 >> v1:172.21.15.134:6790/0 conn(0x7f15440a98f0 legacy=0x7f15441d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:17.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 >> v1:172.21.15.159:6789/0 conn(0x7f154413f710 legacy=0x7f15441d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:17.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:17.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 1 monclient: found mon.a 2024-03-25T19:38:17.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15441d5000 con 0x7f15441430c0 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 10 monclient: _finish_auth 0 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f153affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 5 monclient: authenticate success, global_id 4692 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 10 monclient: _renew_subs 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1544138d40 con 0x7f15441430c0 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 10 monclient: _renew_subs 2024-03-25T19:38:17.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:17.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15441d43c0 con 0x7f15441430c0 2024-03-25T19:38:17.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.701+0000 7f1548990640 1 librados: init done 2024-03-25T19:38:17.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.702+0000 7f1548990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:17.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.702+0000 7f1548990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:17.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.702+0000 7f1548990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:17.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:17.702+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15441d7590 con 0x7f15441430c0 2024-03-25T19:38:18.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.020+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f153c0020a0 con 0x7f15441430c0 2024-03-25T19:38:18.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.020+0000 7f153affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.020+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f153c003990 con 0x7f15441430c0 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.020+0000 7f153affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.020+0000 7f153affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.020+0000 7f153affd640 10 monclient: dump: 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:18.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.020+0000 7f154159a640 4 set_mon_vals no callback set 2024-03-25T19:38:18.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.021+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f153c003bb0 con 0x7f15441430c0 2024-03-25T19:38:18.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.022+0000 7f154259c640 10 monclient: get_auth_request con 0x7f152406e060 auth_method 0 2024-03-25T19:38:18.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.022+0000 7f153affd640 10 client.4692.objecter ms_handle_connect 0x7f152406e060 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f153c0548b0 con 0x7f15441430c0 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 10 client.4692.objecter ms_dispatch 0x7f1544000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 3 client.4692.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 3 client.4692.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f153c0503c0 con 0x7f15441430c0 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:18.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.027+0000 7f153affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:18.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.276+0000 7f1548990640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:18.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.276+0000 7f1548990640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:18.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.276+0000 7f1548990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:18.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.276+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15441d78c0 con 0x7f15441430c0 2024-03-25T19:38:18.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.298+0000 7f153affd640 1 -- 172.21.15.134:0/1030658008 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f153c0522f0 con 0x7f15441430c0 2024-03-25T19:38:18.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.298+0000 7f153affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:18.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.298+0000 7f153affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:18.298 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:18.298 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 10 librados: watch_flush enter 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 10 librados: watch_flush exit 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 10 client.4692.objecter successfully canceled tick 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 >> v1:172.21.15.159:6800/1875874887 conn(0x7f152406e060 legacy=0x7f1524070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 10 monclient: shutdown 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:18.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 >> v1:172.21.15.134:6789/0 conn(0x7f15441430c0 legacy=0x7f1544138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:18.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 shutdown_connections 2024-03-25T19:38:18.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 >> 172.21.15.134:0/1030658008 conn(0x7f15440b7ea0 msgr2=0x7f1544074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:18.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 shutdown_connections 2024-03-25T19:38:18.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 1 -- 172.21.15.134:0/1030658008 wait complete. 2024-03-25T19:38:18.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:18.301+0000 7f1548990640 1 librados: shutdown 2024-03-25T19:38:18.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:25.120 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.240+0000 7f841b5cf640 10 monclient: get_monmap_and_config 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.240+0000 7f841b5cf640 10 monclient: build_initial_monmap 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.240+0000 7f841b5cf640 10 monclient: monmap: 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:25.241439+0000 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:25.241439+0000 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:25.240 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.241+0000 7f841b5cf640 1 Processor -- start 2024-03-25T19:38:25.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.241+0000 7f841b5cf640 1 -- start start 2024-03-25T19:38:25.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.241+0000 7f841b5cf640 10 monclient: init 2024-03-25T19:38:25.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f84140a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 10 monclient(hunting): picked mon.noname-b con 0x7f841413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 10 monclient(hunting): picked mon.noname-c con 0x7f84141430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8414142520 con 0x7f84140a98f0 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84140b28a0 con 0x7f84141430c0 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84140b2a30 con 0x7f841413f710 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:25.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.242+0000 7f841b5cf640 10 monclient(hunting): authenticate will time out at 6537.401024s 2024-03-25T19:38:25.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.251+0000 7f8418b6a640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f841413f710 0x7f84141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59952/0 (socket says 172.21.15.134:59952) 2024-03-25T19:38:25.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.251+0000 7f8418b6a640 1 -- 172.21.15.134:0/4274705847 learned_addr learned my addr 172.21.15.134:0/4274705847 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:25.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.265+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527753276 0 0) 0x7f84140b28a0 con 0x7f84141430c0 2024-03-25T19:38:25.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.265+0000 7f840bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:25.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.265+0000 7f840bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:25.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.265+0000 7f840bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:25.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.265+0000 7f840bfff640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:38:25.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.265+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400003520 con 0x7f84141430c0 2024-03-25T19:38:25.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.269+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575102045 0 0) 0x7f84140b2a30 con 0x7f841413f710 2024-03-25T19:38:25.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.269+0000 7f840bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:25.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.269+0000 7f840bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:25.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.269+0000 7f840bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:25.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.269+0000 7f840bfff640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:38:25.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.269+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84140b28a0 con 0x7f841413f710 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606038132 0 0) 0x7f8400003520 con 0x7f84141430c0 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 >> v1:172.21.15.134:6790/0 conn(0x7f841413f710 legacy=0x7f84141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 >> v1:172.21.15.134:6789/0 conn(0x7f84140a98f0 legacy=0x7f84140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 1 monclient: found mon.noname-c 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84140b44c0 con 0x7f84141430c0 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 10 monclient: _finish_auth 0 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f841b5cf640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:38:25.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f841b5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f840c001650 con 0x7f84141430c0 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f840bfff640 10 monclient: dump: 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.270+0000 7f841b5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:25.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f840c0018a0 con 0x7f84141430c0 2024-03-25T19:38:25.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f840bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:25.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f840bfff640 1 -- 172.21.15.134:0/4274705847 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f840c001c30 con 0x7f84141430c0 2024-03-25T19:38:25.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f841b5cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f841b5cf640 4 set_mon_vals no callback set 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f840bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f840bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.277+0000 7f840bfff640 10 monclient: dump: 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:25.278 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.278+0000 7f841b5cf640 10 monclient: shutdown 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.278+0000 7f841b5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.278+0000 7f841b5cf640 1 -- 172.21.15.134:0/4274705847 >> v1:172.21.15.159:6789/0 conn(0x7f84141430c0 legacy=0x7f841414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.278+0000 7f841b5cf640 1 -- 172.21.15.134:0/4274705847 shutdown_connections 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.278+0000 7f841b5cf640 1 -- 172.21.15.134:0/4274705847 >> 172.21.15.134:0/4274705847 conn(0x7f84140b7ea0 msgr2=0x7f84140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.279+0000 7f841b5cf640 1 -- 172.21.15.134:0/4274705847 shutdown_connections 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.279+0000 7f841b5cf640 1 -- 172.21.15.134:0/4274705847 wait complete. 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.279+0000 7f841b5cf640 10 monclient: build_initial_monmap 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 10 monclient: monmap: 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:25.280 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 1 librados: starting msgr at 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 1 librados: starting objecter 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 1 Processor -- start 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 1 -- start start 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 1 librados: setting wanted keys 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 1 librados: calling monclient init 2024-03-25T19:38:25.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.280+0000 7f841b5cf640 10 monclient: init 2024-03-25T19:38:25.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:25.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:25.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f84140a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:25.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 10 monclient(hunting): picked mon.noname-c con 0x7f841413f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:25.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 10 monclient(hunting): picked mon.noname-b con 0x7f84141430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:25.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84140abb70 con 0x7f84140a98f0 2024-03-25T19:38:25.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84140abe50 con 0x7f84141430c0 2024-03-25T19:38:25.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84140ac130 con 0x7f841413f710 2024-03-25T19:38:25.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:25.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.281+0000 7f841b5cf640 10 monclient(hunting): authenticate will time out at 6537.439944s 2024-03-25T19:38:25.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.295+0000 7f8419b6c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f84141430c0 0x7f84141e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:49624/0 (socket says 172.21.15.134:49624) 2024-03-25T19:38:25.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.295+0000 7f8419b6c640 1 -- 172.21.15.134:0/1598079045 learned_addr learned my addr 172.21.15.134:0/1598079045 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:25.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.295+0000 7f8409ffb640 10 client.?.objecter ms_handle_connect 0x7f84141430c0 2024-03-25T19:38:25.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.295+0000 7f8409ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:25.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.299+0000 7f8409ffb640 10 client.?.objecter ms_handle_connect 0x7f84140a98f0 2024-03-25T19:38:25.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.299+0000 7f8409ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:25.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.304+0000 7f8409ffb640 10 client.?.objecter ms_handle_connect 0x7f841413f710 2024-03-25T19:38:25.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.304+0000 7f8409ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:25.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.315+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727614538 0 0) 0x7f84140abb70 con 0x7f84140a98f0 2024-03-25T19:38:25.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.315+0000 7f8409ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:25.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.315+0000 7f8409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:25.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.315+0000 7f8409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:25.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.315+0000 7f8409ffb640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:38:25.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.315+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f8004eb0 con 0x7f84140a98f0 2024-03-25T19:38:25.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.319+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815777209 0 0) 0x7f84140abe50 con 0x7f84141430c0 2024-03-25T19:38:25.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.319+0000 7f8409ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:25.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.319+0000 7f8409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:25.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.319+0000 7f8409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:25.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.319+0000 7f8409ffb640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:38:25.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.319+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84140abb70 con 0x7f84141430c0 2024-03-25T19:38:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.326+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356269602 0 0) 0x7f84140ac130 con 0x7f841413f710 2024-03-25T19:38:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.326+0000 7f8409ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.326+0000 7f8409ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.326+0000 7f8409ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:25.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.326+0000 7f8409ffb640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:38:25.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.326+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84140abe50 con 0x7f841413f710 2024-03-25T19:38:25.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.334+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1088581015 0 0) 0x7f83f8004eb0 con 0x7f84140a98f0 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84140ac130 con 0x7f84140a98f0 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8410001bf0 con 0x7f84140a98f0 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:25.335 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:25.336 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:25.336 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:25.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2279162397 0 0) 0x7f84140abb70 con 0x7f84141430c0 2024-03-25T19:38:25.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83f8004eb0 con 0x7f84141430c0 2024-03-25T19:38:25.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f840c001bb0 con 0x7f84141430c0 2024-03-25T19:38:25.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.335+0000 7f8409ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:25.337 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.346+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1748459326 0 0) 0x7f84140abe50 con 0x7f841413f710 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.346+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84140abb70 con 0x7f841413f710 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.346+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8404001930 con 0x7f841413f710 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.346+0000 7f8409ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.346+0000 7f8409ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.346+0000 7f8409ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:25.346 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:25.347 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:25.347 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:25.347 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:25.347 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 864909714 0 0) 0x7f84140ac130 con 0x7f84140a98f0 2024-03-25T19:38:25.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 >> v1:172.21.15.134:6790/0 conn(0x7f841413f710 legacy=0x7f84140ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:25.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 >> v1:172.21.15.159:6789/0 conn(0x7f84141430c0 legacy=0x7f84141e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:25.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:25.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 1 monclient: found mon.a 2024-03-25T19:38:25.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8414139130 con 0x7f84140a98f0 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f8409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 10 monclient: _renew_subs 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8414137e40 con 0x7f84140a98f0 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 10 monclient: _renew_subs 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8414138c10 con 0x7f84140a98f0 2024-03-25T19:38:25.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.356+0000 7f841b5cf640 1 librados: init done 2024-03-25T19:38:25.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.358+0000 7f841b5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:25.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.358+0000 7f841b5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:25.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.358+0000 7f841b5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:25.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.358+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84140b3ef0 con 0x7f84140a98f0 2024-03-25T19:38:25.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.574+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8410003970 con 0x7f84140a98f0 2024-03-25T19:38:25.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.574+0000 7f8409ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:25.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.574+0000 7f840b7fe640 4 set_mon_vals no callback set 2024-03-25T19:38:25.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.653+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8410002100 con 0x7f84140a98f0 2024-03-25T19:38:25.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.653+0000 7f8409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:25.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.653+0000 7f8409ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:25.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.653+0000 7f8409ffb640 10 monclient: dump: 2024-03-25T19:38:25.653 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:25.653 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:25.653 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:25.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.654+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f8410019b70 con 0x7f84140a98f0 2024-03-25T19:38:25.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.655+0000 7f8418b6a640 10 monclient: get_auth_request con 0x7f83f806e060 auth_method 0 2024-03-25T19:38:25.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.655+0000 7f8409ffb640 10 client.4695.objecter ms_handle_connect 0x7f83f806e060 2024-03-25T19:38:25.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1777102561 0 0) 0x7f84100539a0 con 0x7f84140a98f0 2024-03-25T19:38:25.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 10 client.4695.objecter ms_dispatch 0x7f8414000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:25.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 3 client.4695.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:25.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 3 client.4695.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:25.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:38:25.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f841004f2e0 con 0x7f84140a98f0 2024-03-25T19:38:25.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:25.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.660+0000 7f8409ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.874+0000 7f841b5cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:25.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.874+0000 7f841b5cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:25.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.874+0000 7f841b5cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:25.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.874+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84140627b0 con 0x7f84140a98f0 2024-03-25T19:38:25.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.905+0000 7f8409ffb640 1 -- 172.21.15.134:0/1598079045 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 611047322) 0x7f84100513e0 con 0x7f84140a98f0 2024-03-25T19:38:25.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.905+0000 7f8409ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:25.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.905+0000 7f8409ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:25.905 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:25.905 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:28:27.749183+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:28:27.749183+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.907+0000 7f841b5cf640 10 librados: watch_flush enter 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.907+0000 7f841b5cf640 10 librados: watch_flush exit 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.907+0000 7f841b5cf640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.907+0000 7f841b5cf640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 >> v1:172.21.15.159:6800/1875874887 conn(0x7f83f806e060 legacy=0x7f83f8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 10 monclient: shutdown 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 >> v1:172.21.15.134:6789/0 conn(0x7f84140a98f0 legacy=0x7f8414136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 shutdown_connections 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 >> 172.21.15.134:0/1598079045 conn(0x7f84140b7ea0 msgr2=0x7f8414074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:25.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 shutdown_connections 2024-03-25T19:38:25.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 1 -- 172.21.15.134:0/1598079045 wait complete. 2024-03-25T19:38:25.909 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:25.908+0000 7f841b5cf640 1 librados: shutdown 2024-03-25T19:38:25.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:31.519 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:36.191 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.571 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:36.243 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.598 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:36.271 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.599 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:36.271 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.620 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:36.292 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.629 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:36.276 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.644 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:36.291 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.649 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:36.322 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:31.683 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:36.330 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:32.726 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:32.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.828+0000 7f97ddc9a640 10 monclient: get_monmap_and_config 2024-03-25T19:38:32.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.828+0000 7f97ddc9a640 10 monclient: build_initial_monmap 2024-03-25T19:38:32.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.828+0000 7f97ddc9a640 10 monclient: monmap: 2024-03-25T19:38:32.827 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:32.827 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:32.829077+0000 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:32.829077+0000 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:32.828 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.828+0000 7f97ddc9a640 1 Processor -- start 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.828+0000 7f97ddc9a640 1 -- start start 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.828+0000 7f97ddc9a640 10 monclient: init 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 10 monclient(hunting): picked mon.noname-a con 0x7f97d80b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 10 monclient(hunting): picked mon.noname-b con 0x7f97d80b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:32.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 10 monclient(hunting): picked mon.noname-c con 0x7f97d814be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:32.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d80af940 con 0x7f97d80b4d60 2024-03-25T19:38:32.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d80b0a70 con 0x7f97d814be90 2024-03-25T19:38:32.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d80b0d50 con 0x7f97d80b1730 2024-03-25T19:38:32.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:32.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.829+0000 7f97ddc9a640 10 monclient(hunting): authenticate will time out at 6544.988086s 2024-03-25T19:38:32.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.830+0000 7f97d6ffd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f97d80b1730 0x7f97d80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:48680/0 (socket says 172.21.15.134:48680) 2024-03-25T19:38:32.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.830+0000 7f97d6ffd640 1 -- 172.21.15.134:0/2689977896 learned_addr learned my addr 172.21.15.134:0/2689977896 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:32.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.863+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096458203 0 0) 0x7f97d80af940 con 0x7f97d80b4d60 2024-03-25T19:38:32.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.863+0000 7f97d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:32.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.863+0000 7f97d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:32.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.863+0000 7f97d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:32.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.863+0000 7f97d67fc640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:38:32.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.863+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c4003520 con 0x7f97d80b4d60 2024-03-25T19:38:32.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2311256358 0 0) 0x7f97c4003520 con 0x7f97d80b4d60 2024-03-25T19:38:32.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 >> v1:172.21.15.134:6790/0 conn(0x7f97d80b1730 legacy=0x7f97d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:32.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 >> v1:172.21.15.159:6789/0 conn(0x7f97d814be90 legacy=0x7f97d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 1 monclient: found mon.noname-a 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d80b1030 con 0x7f97d80b4d60 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f97d0001a10 con 0x7f97d80b4d60 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97d67fc640 10 monclient: dump: 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.884 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97ddc9a640 5 monclient: authenticate success, global_id 4698 2024-03-25T19:38:32.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.883+0000 7f97ddc9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:32.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97d0001b70 con 0x7f97d80b4d60 2024-03-25T19:38:32.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:32.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97d67fc640 1 -- 172.21.15.134:0/2689977896 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f97d0001fc0 con 0x7f97d80b4d60 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97ddc9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97ddc9a640 4 set_mon_vals no callback set 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.894+0000 7f97d67fc640 10 monclient: dump: 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:32.895 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:32.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.895+0000 7f97ddc9a640 10 monclient: shutdown 2024-03-25T19:38:32.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.895+0000 7f97ddc9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.895+0000 7f97ddc9a640 1 -- 172.21.15.134:0/2689977896 >> v1:172.21.15.134:6789/0 conn(0x7f97d80b4d60 legacy=0x7f97d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.895+0000 7f97ddc9a640 1 -- 172.21.15.134:0/2689977896 shutdown_connections 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.895+0000 7f97ddc9a640 1 -- 172.21.15.134:0/2689977896 >> 172.21.15.134:0/2689977896 conn(0x7f97d80617a0 msgr2=0x7f97d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.895+0000 7f97ddc9a640 1 -- 172.21.15.134:0/2689977896 shutdown_connections 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.895+0000 7f97ddc9a640 1 -- 172.21.15.134:0/2689977896 wait complete. 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.896+0000 7f97ddc9a640 10 monclient: build_initial_monmap 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.896+0000 7f97ddc9a640 10 monclient: monmap: 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:32.897 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:32.898 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.897+0000 7f97ddc9a640 1 librados: starting msgr at 2024-03-25T19:38:32.898 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.897+0000 7f97ddc9a640 1 librados: starting objecter 2024-03-25T19:38:32.898 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.897+0000 7f97ddc9a640 1 Processor -- start 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.897+0000 7f97ddc9a640 1 -- start start 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.897+0000 7f97ddc9a640 1 librados: setting wanted keys 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.897+0000 7f97ddc9a640 1 librados: calling monclient init 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.897+0000 7f97ddc9a640 10 monclient: init 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 10 monclient(hunting): picked mon.noname-c con 0x7f97d80b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 10 monclient(hunting): picked mon.noname-a con 0x7f97d80b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 10 monclient(hunting): picked mon.noname-b con 0x7f97d814be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d81d7d80 con 0x7f97d80b4d60 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d81d78a0 con 0x7f97d814be90 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d81d7a30 con 0x7f97d80b1730 2024-03-25T19:38:32.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:32.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.898+0000 7f97ddc9a640 10 monclient(hunting): authenticate will time out at 6545.057335s 2024-03-25T19:38:32.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.915+0000 7f97d7fff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f97d814be90 0x7f97d81369b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58302/0 (socket says 172.21.15.134:58302) 2024-03-25T19:38:32.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.915+0000 7f97d7fff640 1 -- 172.21.15.134:0/1154877235 learned_addr learned my addr 172.21.15.134:0/1154877235 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:32.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.916+0000 7f97cffff640 10 client.?.objecter ms_handle_connect 0x7f97d814be90 2024-03-25T19:38:32.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.916+0000 7f97cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:32.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.917+0000 7f97cffff640 10 client.?.objecter ms_handle_connect 0x7f97d80b4d60 2024-03-25T19:38:32.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.917+0000 7f97cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:32.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.924+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689015488 0 0) 0x7f97d81d78a0 con 0x7f97d814be90 2024-03-25T19:38:32.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.924+0000 7f97cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:32.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.924+0000 7f97cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:32.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.924+0000 7f97cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:32.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.924+0000 7f97cffff640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:38:32.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.924+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b8004eb0 con 0x7f97d814be90 2024-03-25T19:38:32.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 10 client.?.objecter ms_handle_connect 0x7f97d80b1730 2024-03-25T19:38:32.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:32.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960452322 0 0) 0x7f97d81d7d80 con 0x7f97d80b4d60 2024-03-25T19:38:32.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:32.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:32.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:32.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:38:32.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.931+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d81d78a0 con 0x7f97d80b4d60 2024-03-25T19:38:32.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.938+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 294429205 0 0) 0x7f97b8004eb0 con 0x7f97d814be90 2024-03-25T19:38:32.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.938+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97d81d7d80 con 0x7f97d814be90 2024-03-25T19:38:32.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.938+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f97c80018b0 con 0x7f97d814be90 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.938+0000 7f97cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.938+0000 7f97cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.938+0000 7f97cffff640 10 monclient(hunting): dump: 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:32.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.938+0000 7f97cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:32.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.943+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857283692 0 0) 0x7f97d81d7a30 con 0x7f97d80b1730 2024-03-25T19:38:32.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.943+0000 7f97cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:32.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.943+0000 7f97cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:32.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.943+0000 7f97cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:32.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.943+0000 7f97cffff640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:38:32.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.943+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b8004eb0 con 0x7f97d80b1730 2024-03-25T19:38:32.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.946+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3667840389 0 0) 0x7f97d81d78a0 con 0x7f97d80b4d60 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.946+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97d81d7a30 con 0x7f97d80b4d60 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.946+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f97c00018a0 con 0x7f97d80b4d60 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.946+0000 7f97cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.946+0000 7f97cffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.946+0000 7f97cffff640 10 monclient(hunting): dump: 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:32.946 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:32.947 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:32.947 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.956+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 880899641 0 0) 0x7f97b8004eb0 con 0x7f97d80b1730 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.956+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97d81d78a0 con 0x7f97d80b1730 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.956+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f97d0001f40 con 0x7f97d80b1730 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.956+0000 7f97cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.956+0000 7f97cffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.956+0000 7f97cffff640 10 monclient(hunting): dump: 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:32.956 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:32.957 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:32.957 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:32.957 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:32.957 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:32.957 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 758255841 0 0) 0x7f97d81d7d80 con 0x7f97d814be90 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 >> v1:172.21.15.134:6790/0 conn(0x7f97d80b1730 legacy=0x7f97d81d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 >> v1:172.21.15.134:6789/0 conn(0x7f97d80b4d60 legacy=0x7f97d8143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 1 monclient: found mon.b 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8136200 con 0x7f97d814be90 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 10 monclient: _finish_auth 0 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.958+0000 7f97ddc9a640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:38:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 10 monclient: _renew_subs 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d81364d0 con 0x7f97d814be90 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 10 monclient: _renew_subs 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d8130040 con 0x7f97d814be90 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.959+0000 7f97ddc9a640 1 librados: init done 2024-03-25T19:38:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.960+0000 7f97ddc9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:32.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.960+0000 7f97ddc9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:32.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.960+0000 7f97ddc9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:32.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:32.960+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d8138340 con 0x7f97d814be90 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.253+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97c8003610 con 0x7f97d814be90 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.253+0000 7f97cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.253+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f97c80039a0 con 0x7f97d814be90 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.253+0000 7f97cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.253+0000 7f97cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.253+0000 7f97cffff640 10 monclient: dump: 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:33.253 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:33.254 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:33.254 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:33.254 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:33.254 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:33.254 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:33.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.253+0000 7f97d5ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:33.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.254+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f97c8003b80 con 0x7f97d814be90 2024-03-25T19:38:33.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.255+0000 7f97d77fe640 10 monclient: get_auth_request con 0x7f97b806e110 auth_method 0 2024-03-25T19:38:33.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.255+0000 7f97cffff640 10 client.4663.objecter ms_handle_connect 0x7f97b806e110 2024-03-25T19:38:33.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.259+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7f97c8054600 con 0x7f97d814be90 2024-03-25T19:38:33.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.259+0000 7f97cffff640 10 client.4663.objecter ms_dispatch 0x7f97d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:33.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.259+0000 7f97cffff640 3 client.4663.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:33.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.259+0000 7f97cffff640 3 client.4663.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:33.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.259+0000 7f97cffff640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:38:33.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.261+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97c8050110 con 0x7f97d814be90 2024-03-25T19:38:33.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.261+0000 7f97cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:33.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.261+0000 7f97cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:33.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.511+0000 7f97ddc9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:33.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.511+0000 7f97ddc9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:33.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.511+0000 7f97ddc9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:33.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.511+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97d80627b0 con 0x7f97d814be90 2024-03-25T19:38:33.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.552+0000 7f97cffff640 1 -- 172.21.15.134:0/1154877235 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7f97c8052040 con 0x7f97d814be90 2024-03-25T19:38:33.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.552+0000 7f97cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:33.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.552+0000 7f97cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:33.552 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:33.552 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:33.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 10 librados: watch_flush enter 2024-03-25T19:38:33.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 10 librados: watch_flush exit 2024-03-25T19:38:33.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:38:33.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:38:33.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 >> v1:172.21.15.159:6800/1875874887 conn(0x7f97b806e110 legacy=0x7f97b80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:33.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 10 monclient: shutdown 2024-03-25T19:38:33.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:33.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 >> v1:172.21.15.159:6789/0 conn(0x7f97d814be90 legacy=0x7f97d81369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:33.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 shutdown_connections 2024-03-25T19:38:33.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 >> 172.21.15.134:0/1154877235 conn(0x7f97d80617a0 msgr2=0x7f97d80b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:33.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 shutdown_connections 2024-03-25T19:38:33.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 1 -- 172.21.15.134:0/1154877235 wait complete. 2024-03-25T19:38:33.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:33.555+0000 7f97ddc9a640 1 librados: shutdown 2024-03-25T19:38:33.567 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:40.374 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:40.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.490+0000 7f1a0d332640 10 monclient: get_monmap_and_config 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.490+0000 7f1a0d332640 10 monclient: build_initial_monmap 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.490+0000 7f1a0d332640 10 monclient: monmap: 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:40.491112+0000 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:40.491112+0000 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:40.490 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.490+0000 7f1a0d332640 1 Processor -- start 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.490+0000 7f1a0d332640 1 -- start start 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.490+0000 7f1a0d332640 10 monclient: init 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 10 monclient(hunting): picked mon.noname-c con 0x7f1a080a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 10 monclient(hunting): picked mon.noname-b con 0x7f1a0813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 10 monclient(hunting): picked mon.noname-a con 0x7f1a081430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a08142520 con 0x7f1a081430c0 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a080b28a0 con 0x7f1a080a98f0 2024-03-25T19:38:40.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a080b2a30 con 0x7f1a0813f710 2024-03-25T19:38:40.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:40.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a0d332640 10 monclient(hunting): authenticate will time out at 6552.650208s 2024-03-25T19:38:40.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.491+0000 7f1a06ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f1a080a98f0 0x7f1a080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34736/0 (socket says 172.21.15.134:34736) 2024-03-25T19:38:40.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.492+0000 7f1a06ffd640 1 -- 172.21.15.134:0/2074668949 learned_addr learned my addr 172.21.15.134:0/2074668949 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:40.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.568+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107115141 0 0) 0x7f1a08142520 con 0x7f1a081430c0 2024-03-25T19:38:40.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.568+0000 7f1a067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:40.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.568+0000 7f1a067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:40.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.568+0000 7f1a067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:40.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.568+0000 7f1a067fc640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:38:40.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.568+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f4003520 con 0x7f1a081430c0 2024-03-25T19:38:40.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 917839846 0 0) 0x7f19f4003520 con 0x7f1a081430c0 2024-03-25T19:38:40.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 >> v1:172.21.15.134:6790/0 conn(0x7f1a0813f710 legacy=0x7f1a081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 >> v1:172.21.15.159:6789/0 conn(0x7f1a080a98f0 legacy=0x7f1a080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 1 monclient: found mon.noname-a 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080b44c0 con 0x7f1a081430c0 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1a00001a10 con 0x7f1a081430c0 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a0d332640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a0d332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.594+0000 7f1a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.595+0000 7f1a067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:40.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.595+0000 7f1a067fc640 10 monclient: dump: 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:40.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.595+0000 7f1a0d332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a00001b70 con 0x7f1a081430c0 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a067fc640 1 -- 172.21.15.134:0/2074668949 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1a00001f80 con 0x7f1a081430c0 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a0d332640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a0d332640 4 set_mon_vals no callback set 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.619+0000 7f1a067fc640 10 monclient: dump: 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.619 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.620 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:40.620 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:40.620 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:40.620 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:40.620 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:40.620 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 10 monclient: shutdown 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 1 -- 172.21.15.134:0/2074668949 >> v1:172.21.15.134:6789/0 conn(0x7f1a081430c0 legacy=0x7f1a0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 1 -- 172.21.15.134:0/2074668949 shutdown_connections 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 1 -- 172.21.15.134:0/2074668949 >> 172.21.15.134:0/2074668949 conn(0x7f1a080b7ea0 msgr2=0x7f1a080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 1 -- 172.21.15.134:0/2074668949 shutdown_connections 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 1 -- 172.21.15.134:0/2074668949 wait complete. 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 10 monclient: build_initial_monmap 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.620+0000 7f1a0d332640 10 monclient: monmap: 2024-03-25T19:38:40.622 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.621+0000 7f1a0d332640 1 librados: starting msgr at 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.621+0000 7f1a0d332640 1 librados: starting objecter 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.621+0000 7f1a0d332640 1 Processor -- start 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.621+0000 7f1a0d332640 1 -- start start 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.621+0000 7f1a0d332640 1 librados: setting wanted keys 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.621+0000 7f1a0d332640 1 librados: calling monclient init 2024-03-25T19:38:40.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.621+0000 7f1a0d332640 10 monclient: init 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 10 monclient(hunting): picked mon.noname-b con 0x7f1a080a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 10 monclient(hunting): picked mon.noname-c con 0x7f1a0813f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 10 monclient(hunting): picked mon.noname-a con 0x7f1a081430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a08139a10 con 0x7f1a081430c0 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a081d7880 con 0x7f1a080a98f0 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a08137890 con 0x7f1a0813f710 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:40.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.622+0000 7f1a0d332640 10 monclient(hunting): authenticate will time out at 6552.780801s 2024-03-25T19:38:40.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.633+0000 7f1a06ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f1a080a98f0 0x7f1a081d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58510/0 (socket says 172.21.15.134:58510) 2024-03-25T19:38:40.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.633+0000 7f1a06ffd640 1 -- 172.21.15.134:0/2806621925 learned_addr learned my addr 172.21.15.134:0/2806621925 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:40.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.634+0000 7f19ff7fe640 10 client.?.objecter ms_handle_connect 0x7f1a080a98f0 2024-03-25T19:38:40.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.634+0000 7f19ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:40.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.636+0000 7f19ff7fe640 10 client.?.objecter ms_handle_connect 0x7f1a0813f710 2024-03-25T19:38:40.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.636+0000 7f19ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:40.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.646+0000 7f19ff7fe640 10 client.?.objecter ms_handle_connect 0x7f1a081430c0 2024-03-25T19:38:40.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.646+0000 7f19ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:40.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.713+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041143913 0 0) 0x7f1a08139a10 con 0x7f1a081430c0 2024-03-25T19:38:40.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.713+0000 7f19ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:40.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.713+0000 7f19ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:40.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.713+0000 7f19ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:40.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.713+0000 7f19ff7fe640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:38:40.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.713+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e8004eb0 con 0x7f1a081430c0 2024-03-25T19:38:40.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.718+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854160859 0 0) 0x7f1a08137890 con 0x7f1a0813f710 2024-03-25T19:38:40.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.718+0000 7f19ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:40.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.718+0000 7f19ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:40.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.718+0000 7f19ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:40.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.718+0000 7f19ff7fe640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:38:40.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.718+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a08139a10 con 0x7f1a0813f710 2024-03-25T19:38:40.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.719+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724942897 0 0) 0x7f1a081d7880 con 0x7f1a080a98f0 2024-03-25T19:38:40.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.719+0000 7f19ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:40.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.719+0000 7f19ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:40.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.719+0000 7f19ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:40.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.719+0000 7f19ff7fe640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:38:40.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.719+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a08137890 con 0x7f1a080a98f0 2024-03-25T19:38:40.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.732+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4167353054 0 0) 0x7f1a08139a10 con 0x7f1a0813f710 2024-03-25T19:38:40.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.732+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a081d7880 con 0x7f1a0813f710 2024-03-25T19:38:40.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.732+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f19f00018a0 con 0x7f1a0813f710 2024-03-25T19:38:40.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.732+0000 7f19ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.732+0000 7f19ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.732+0000 7f19ff7fe640 10 monclient(hunting): dump: 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:40.733 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.732+0000 7f19ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:40.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.734+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3178814810 0 0) 0x7f19e8004eb0 con 0x7f1a081430c0 2024-03-25T19:38:40.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.734+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a08139a10 con 0x7f1a081430c0 2024-03-25T19:38:40.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.734+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1a00001f40 con 0x7f1a081430c0 2024-03-25T19:38:40.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.734+0000 7f19ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:40.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.734+0000 7f19ff7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:40.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.734+0000 7f19ff7fe640 10 monclient(hunting): dump: 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:40.735 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:40.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3503487089 0 0) 0x7f1a08139a10 con 0x7f1a081430c0 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 >> v1:172.21.15.134:6790/0 conn(0x7f1a0813f710 legacy=0x7f1a08138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 >> v1:172.21.15.159:6789/0 conn(0x7f1a080a98f0 legacy=0x7f1a081d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 1 monclient: found mon.a 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a081d84b0 con 0x7f1a081430c0 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f19ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 10 monclient: _renew_subs 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a081d51f0 con 0x7f1a081430c0 2024-03-25T19:38:40.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:40.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 10 monclient: _renew_subs 2024-03-25T19:38:40.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:40.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a08138c60 con 0x7f1a081430c0 2024-03-25T19:38:40.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.741+0000 7f1a0d332640 1 librados: init done 2024-03-25T19:38:40.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.742+0000 7f1a0d332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:40.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.743+0000 7f1a0d332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:40.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.743+0000 7f1a0d332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:40.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:40.743+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a080627b0 con 0x7f1a081430c0 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.046+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a00003c20 con 0x7f1a081430c0 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.047+0000 7f19ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.047+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1a00003f30 con 0x7f1a081430c0 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.047+0000 7f19ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.047+0000 7f19ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.047+0000 7f19ff7fe640 10 monclient: dump: 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:41.047 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:41.048 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:41.048 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:41.048 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:41.048 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:41.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.047+0000 7f1a05ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:41.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.048+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f1a00019f00 con 0x7f1a081430c0 2024-03-25T19:38:41.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.048+0000 7f1a06ffd640 10 monclient: get_auth_request con 0x7f19e806dfb0 auth_method 0 2024-03-25T19:38:41.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.048+0000 7f19ff7fe640 10 client.4707.objecter ms_handle_connect 0x7f19e806dfb0 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7f1a000539c0 con 0x7f1a081430c0 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 10 client.4707.objecter ms_dispatch 0x7f1a08000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a0004f4d0 con 0x7f1a081430c0 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:41.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.053+0000 7f19ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:41.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.303+0000 7f1a0d332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:41.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.303+0000 7f1a0d332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:41.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.303+0000 7f1a0d332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:38:41.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.303+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a0813df90 con 0x7f1a081430c0 2024-03-25T19:38:41.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.329+0000 7f19ff7fe640 1 -- 172.21.15.134:0/2806621925 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7f1a00051400 con 0x7f1a081430c0 2024-03-25T19:38:41.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.329+0000 7f19ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:41.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.329+0000 7f19ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:41.329 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:41.329 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.331+0000 7f1a0d332640 10 librados: watch_flush enter 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.331+0000 7f1a0d332640 10 librados: watch_flush exit 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.331+0000 7f1a0d332640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.331+0000 7f1a0d332640 10 client.4707.objecter successfully canceled tick 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.331+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 >> v1:172.21.15.159:6800/1875874887 conn(0x7f19e806dfb0 legacy=0x7f19e8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 10 monclient: shutdown 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 >> v1:172.21.15.134:6789/0 conn(0x7f1a081430c0 legacy=0x7f1a081e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 shutdown_connections 2024-03-25T19:38:41.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 >> 172.21.15.134:0/2806621925 conn(0x7f1a080b7ea0 msgr2=0x7f1a08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:41.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 shutdown_connections 2024-03-25T19:38:41.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 1 -- 172.21.15.134:0/2806621925 wait complete. 2024-03-25T19:38:41.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:41.332+0000 7f1a0d332640 1 librados: shutdown 2024-03-25T19:38:41.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:48.151 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:48.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.265+0000 7fc01a855640 10 monclient: get_monmap_and_config 2024-03-25T19:38:48.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.265+0000 7fc01a855640 10 monclient: build_initial_monmap 2024-03-25T19:38:48.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.265+0000 7fc01a855640 10 monclient: monmap: 2024-03-25T19:38:48.265 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:48.266969+0000 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:48.266969+0000 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:48.266 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.266+0000 7fc01a855640 1 Processor -- start 2024-03-25T19:38:48.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.266+0000 7fc01a855640 1 -- start start 2024-03-25T19:38:48.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.266+0000 7fc01a855640 10 monclient: init 2024-03-25T19:38:48.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 10 monclient(hunting): picked mon.noname-c con 0x7fc0140b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 10 monclient(hunting): picked mon.noname-a con 0x7fc0140b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 10 monclient(hunting): picked mon.noname-b con 0x7fc01414be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140af940 con 0x7fc0140b1730 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140b0a70 con 0x7fc0140b4d60 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140b0d50 con 0x7fc01414be90 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:48.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.267+0000 7fc01a855640 10 monclient(hunting): authenticate will time out at 6560.426305s 2024-03-25T19:38:48.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.273+0000 7fc0137fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc0140b1730 0x7fc0140b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34104/0 (socket says 172.21.15.134:34104) 2024-03-25T19:38:48.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.273+0000 7fc0137fe640 1 -- 172.21.15.134:0/4059803774 learned_addr learned my addr 172.21.15.134:0/4059803774 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:48.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.290+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324846252 0 0) 0x7fc0140af940 con 0x7fc0140b1730 2024-03-25T19:38:48.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.290+0000 7fc012ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:48.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.290+0000 7fc012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:48.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.290+0000 7fc012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:48.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.290+0000 7fc012ffd640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:38:48.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.290+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbffc003520 con 0x7fc0140b1730 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170703460 0 0) 0x7fbffc003520 con 0x7fc0140b1730 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 >> v1:172.21.15.134:6790/0 conn(0x7fc01414be90 legacy=0x7fc01414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 >> v1:172.21.15.159:6789/0 conn(0x7fc0140b4d60 legacy=0x7fc01413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 1 monclient: found mon.noname-a 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0140b1030 con 0x7fc0140b1730 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.301+0000 7fc012ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc012ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:48.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc008001690 con 0x7fc0140b1730 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc012ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc012ffd640 10 monclient: dump: 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc01a855640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:38:48.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.302+0000 7fc01a855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0080018b0 con 0x7fc0140b1730 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc012ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc012ffd640 1 -- 172.21.15.134:0/4059803774 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc008001c40 con 0x7fc0140b1730 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc01a855640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc01a855640 4 set_mon_vals no callback set 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc012ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.316+0000 7fc012ffd640 10 monclient: dump: 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:48.316 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:48.317 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.317+0000 7fc01a855640 10 monclient: shutdown 2024-03-25T19:38:48.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.317+0000 7fc01a855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:48.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.317+0000 7fc01a855640 1 -- 172.21.15.134:0/4059803774 >> v1:172.21.15.134:6789/0 conn(0x7fc0140b1730 legacy=0x7fc0140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.317+0000 7fc01a855640 1 -- 172.21.15.134:0/4059803774 shutdown_connections 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.317+0000 7fc01a855640 1 -- 172.21.15.134:0/4059803774 >> 172.21.15.134:0/4059803774 conn(0x7fc0140617a0 msgr2=0x7fc014061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.317+0000 7fc01a855640 1 -- 172.21.15.134:0/4059803774 shutdown_connections 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.317+0000 7fc01a855640 1 -- 172.21.15.134:0/4059803774 wait complete. 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 10 monclient: build_initial_monmap 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 10 monclient: monmap: 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:48.320 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 1 librados: starting msgr at 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 1 librados: starting objecter 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 1 Processor -- start 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 1 -- start start 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 1 librados: setting wanted keys 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 1 librados: calling monclient init 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.318+0000 7fc01a855640 10 monclient: init 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 10 monclient(hunting): picked mon.noname-b con 0x7fc0140b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 10 monclient(hunting): picked mon.noname-a con 0x7fc0140b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 10 monclient(hunting): picked mon.noname-c con 0x7fc01414be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc014136e40 con 0x7fc0140b4d60 2024-03-25T19:38:48.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0141307c0 con 0x7fc0140b1730 2024-03-25T19:38:48.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc014138610 con 0x7fc01414be90 2024-03-25T19:38:48.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:48.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.319+0000 7fc01a855640 10 monclient(hunting): authenticate will time out at 6560.478242s 2024-03-25T19:38:48.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.326+0000 7fc018df2640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc01414be90 0x7fc0141300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46694/0 (socket says 172.21.15.134:46694) 2024-03-25T19:38:48.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.326+0000 7fc018df2640 1 -- 172.21.15.134:0/1410981117 learned_addr learned my addr 172.21.15.134:0/1410981117 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:48.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.327+0000 7fc010ff9640 10 client.?.objecter ms_handle_connect 0x7fc01414be90 2024-03-25T19:38:48.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.327+0000 7fc010ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:48.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.335+0000 7fc010ff9640 10 client.?.objecter ms_handle_connect 0x7fc0140b1730 2024-03-25T19:38:48.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.335+0000 7fc010ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:48.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.336+0000 7fc010ff9640 10 client.?.objecter ms_handle_connect 0x7fc0140b4d60 2024-03-25T19:38:48.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.336+0000 7fc010ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:48.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.340+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968070703 0 0) 0x7fc014138610 con 0x7fc01414be90 2024-03-25T19:38:48.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.340+0000 7fc010ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:48.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.340+0000 7fc010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:48.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.340+0000 7fc010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:48.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.340+0000 7fc010ff9640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:38:48.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.340+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc000004eb0 con 0x7fc01414be90 2024-03-25T19:38:48.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.344+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706756966 0 0) 0x7fc0141307c0 con 0x7fc0140b1730 2024-03-25T19:38:48.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.344+0000 7fc010ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:48.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.344+0000 7fc010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:48.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.344+0000 7fc010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:48.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.344+0000 7fc010ff9640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:38:48.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.344+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014138610 con 0x7fc0140b1730 2024-03-25T19:38:48.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.345+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766837369 0 0) 0x7fc014136e40 con 0x7fc0140b4d60 2024-03-25T19:38:48.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.345+0000 7fc010ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:48.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.345+0000 7fc010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:48.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.345+0000 7fc010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:48.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.345+0000 7fc010ff9640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:38:48.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.345+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0141307c0 con 0x7fc0140b4d60 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.356+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1117966511 0 0) 0x7fc000004eb0 con 0x7fc01414be90 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.356+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc014136e40 con 0x7fc01414be90 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.356+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc00c001c90 con 0x7fc01414be90 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.356+0000 7fc010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.356+0000 7fc010ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.356+0000 7fc010ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:48.356 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.356+0000 7fc010ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:48.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.362+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 781599930 0 0) 0x7fc014138610 con 0x7fc0140b1730 2024-03-25T19:38:48.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.362+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc000004eb0 con 0x7fc0140b1730 2024-03-25T19:38:48.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.362+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc0040018b0 con 0x7fc0140b1730 2024-03-25T19:38:48.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.362+0000 7fc010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:48.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.362+0000 7fc010ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:48.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.362+0000 7fc010ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:48.363 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.367+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 60860619 0 0) 0x7fc0141307c0 con 0x7fc0140b4d60 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.367+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc014138610 con 0x7fc0140b4d60 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.367+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc0080015d0 con 0x7fc0140b4d60 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.367+0000 7fc010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.367+0000 7fc010ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.367+0000 7fc010ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:48.367 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:48.368 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3420271920 0 0) 0x7fc000004eb0 con 0x7fc0140b1730 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 >> v1:172.21.15.134:6790/0 conn(0x7fc01414be90 legacy=0x7fc0141300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 >> v1:172.21.15.134:6789/0 conn(0x7fc0140b4d60 legacy=0x7fc01412f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 1 monclient: found mon.b 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0141388f0 con 0x7fc0140b1730 2024-03-25T19:38:48.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc010ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 10 monclient: _renew_subs 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0141410e0 con 0x7fc0140b1730 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 10 monclient: _renew_subs 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0141381a0 con 0x7fc0140b1730 2024-03-25T19:38:48.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.373+0000 7fc01a855640 1 librados: init done 2024-03-25T19:38:48.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.374+0000 7fc01a855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:48.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.374+0000 7fc01a855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:48.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.374+0000 7fc01a855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:48.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.374+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0141401f0 con 0x7fc0140b1730 2024-03-25T19:38:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.648+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0040035f0 con 0x7fc0140b1730 2024-03-25T19:38:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.648+0000 7fc010ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.648+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fc004003900 con 0x7fc0140b1730 2024-03-25T19:38:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.648+0000 7fc010ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.648+0000 7fc010ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.648+0000 7fc010ff9640 10 monclient: dump: 2024-03-25T19:38:48.648 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:48.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.648+0000 7fc0127fc640 4 set_mon_vals no callback set 2024-03-25T19:38:48.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.649+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fc0040198d0 con 0x7fc0140b1730 2024-03-25T19:38:48.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.649+0000 7fc0137fe640 10 monclient: get_auth_request con 0x7fc00006e0c0 auth_method 0 2024-03-25T19:38:48.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.650+0000 7fc010ff9640 10 client.4675.objecter ms_handle_connect 0x7fc00006e0c0 2024-03-25T19:38:48.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.653+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7fc0040535c0 con 0x7fc0140b1730 2024-03-25T19:38:48.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.653+0000 7fc010ff9640 10 client.4675.objecter ms_dispatch 0x7fc014000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:48.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.653+0000 7fc010ff9640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:48.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.653+0000 7fc010ff9640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:48.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.653+0000 7fc010ff9640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T19:38:48.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.654+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc00404ef00 con 0x7fc0140b1730 2024-03-25T19:38:48.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.654+0000 7fc010ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:48.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.654+0000 7fc010ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:48.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.868+0000 7fc01a855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:48.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.868+0000 7fc01a855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:48.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.868+0000 7fc01a855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:38:48.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.868+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0140627b0 con 0x7fc0140b1730 2024-03-25T19:38:48.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.917+0000 7fc010ff9640 1 -- 172.21.15.134:0/1410981117 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7fc004051000 con 0x7fc0140b1730 2024-03-25T19:38:48.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.917+0000 7fc010ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:48.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.917+0000 7fc010ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:48.917 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:48.917 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:48.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 10 librados: watch_flush enter 2024-03-25T19:38:48.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 10 librados: watch_flush exit 2024-03-25T19:38:48.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T19:38:48.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 10 client.4675.objecter successfully canceled tick 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 >> v1:172.21.15.159:6800/1875874887 conn(0x7fc00006e0c0 legacy=0x7fc000070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 10 monclient: shutdown 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 >> v1:172.21.15.159:6789/0 conn(0x7fc0140b1730 legacy=0x7fc0141409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 shutdown_connections 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 >> 172.21.15.134:0/1410981117 conn(0x7fc0140617a0 msgr2=0x7fc0140b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 shutdown_connections 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 1 -- 172.21.15.134:0/1410981117 wait complete. 2024-03-25T19:38:48.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:48.919+0000 7fc01a855640 1 librados: shutdown 2024-03-25T19:38:48.930 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:51.520 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:56.193 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.537 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:56.210 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.571 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:56.243 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.574 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:56.247 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.577 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:56.224 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.581 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:56.228 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.583 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:56.256 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.606 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:56.253 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:51.613 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:56.285 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:55.737 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:55.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.853+0000 7f36f7920640 10 monclient: get_monmap_and_config 2024-03-25T19:38:55.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.853+0000 7f36f7920640 10 monclient: build_initial_monmap 2024-03-25T19:38:55.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.853+0000 7f36f7920640 10 monclient: monmap: 2024-03-25T19:38:55.853 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:55.853 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:38:55.854854+0000 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:38:55.854854+0000 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:38:55.854 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.854+0000 7f36f7920640 1 Processor -- start 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.854+0000 7f36f7920640 1 -- start start 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.854+0000 7f36f7920640 10 monclient: init 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 10 monclient(hunting): picked mon.noname-b con 0x7f36f00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 10 monclient(hunting): picked mon.noname-a con 0x7f36f013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 10 monclient(hunting): picked mon.noname-c con 0x7f36f01430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f0142520 con 0x7f36f013f710 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f00b28a0 con 0x7f36f01430c0 2024-03-25T19:38:55.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f00b2a30 con 0x7f36f00a98f0 2024-03-25T19:38:55.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:55.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.855+0000 7f36f7920640 10 monclient(hunting): authenticate will time out at 6568.014061s 2024-03-25T19:38:55.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.861+0000 7f36f56bc640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f36f00a98f0 0x7f36f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:56012/0 (socket says 172.21.15.134:56012) 2024-03-25T19:38:55.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.861+0000 7f36f56bc640 1 -- 172.21.15.134:0/4181915500 learned_addr learned my addr 172.21.15.134:0/4181915500 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:55.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.870+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381537280 0 0) 0x7f36f00b2a30 con 0x7f36f00a98f0 2024-03-25T19:38:55.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.870+0000 7f36e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:55.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.870+0000 7f36e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:55.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.870+0000 7f36e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:55.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.870+0000 7f36e7fff640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:38:55.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.870+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36dc003520 con 0x7f36f00a98f0 2024-03-25T19:38:55.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.876+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308633428 0 0) 0x7f36f00b28a0 con 0x7f36f01430c0 2024-03-25T19:38:55.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.876+0000 7f36e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:55.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.876+0000 7f36e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:55.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.876+0000 7f36e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:55.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.876+0000 7f36e7fff640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:38:55.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.876+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f00b2a30 con 0x7f36f01430c0 2024-03-25T19:38:55.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.882+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939502872 0 0) 0x7f36f0142520 con 0x7f36f013f710 2024-03-25T19:38:55.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.882+0000 7f36e7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:55.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.882+0000 7f36e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:55.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.882+0000 7f36e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:55.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.882+0000 7f36e7fff640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:38:55.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.882+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f00b28a0 con 0x7f36f013f710 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2642376665 0 0) 0x7f36f00b2a30 con 0x7f36f01430c0 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 >> v1:172.21.15.134:6790/0 conn(0x7f36f00a98f0 legacy=0x7f36f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 >> v1:172.21.15.134:6789/0 conn(0x7f36f013f710 legacy=0x7f36f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 1 monclient: found mon.noname-c 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f00b44c0 con 0x7f36f01430c0 2024-03-25T19:38:55.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f36e80019f0 con 0x7f36f01430c0 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36e7fff640 10 monclient: dump: 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:55.886 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:55.887 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:55.887 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:55.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36f7920640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:38:55.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.885+0000 7f36f7920640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e8001b50 con 0x7f36f01430c0 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36e7fff640 1 -- 172.21.15.134:0/4181915500 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f36e8001f60 con 0x7f36f01430c0 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36f7920640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36f7920640 4 set_mon_vals no callback set 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.891+0000 7f36e7fff640 10 monclient: dump: 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:55.891 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:55.892 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:55.892 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:55.892 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:55.892 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:55.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.892+0000 7f36f7920640 10 monclient: shutdown 2024-03-25T19:38:55.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.892+0000 7f36f7920640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:55.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.892+0000 7f36f7920640 1 -- 172.21.15.134:0/4181915500 >> v1:172.21.15.159:6789/0 conn(0x7f36f01430c0 legacy=0x7f36f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:55.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.892+0000 7f36f7920640 1 -- 172.21.15.134:0/4181915500 shutdown_connections 2024-03-25T19:38:55.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.892+0000 7f36f7920640 1 -- 172.21.15.134:0/4181915500 >> 172.21.15.134:0/4181915500 conn(0x7f36f00b7ea0 msgr2=0x7f36f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:55.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.892+0000 7f36f7920640 1 -- 172.21.15.134:0/4181915500 shutdown_connections 2024-03-25T19:38:55.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.892+0000 7f36f7920640 1 -- 172.21.15.134:0/4181915500 wait complete. 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 10 monclient: build_initial_monmap 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 10 monclient: monmap: 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 1 librados: starting msgr at 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 1 librados: starting objecter 2024-03-25T19:38:55.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 1 Processor -- start 2024-03-25T19:38:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 1 -- start start 2024-03-25T19:38:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 1 librados: setting wanted keys 2024-03-25T19:38:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 1 librados: calling monclient init 2024-03-25T19:38:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.893+0000 7f36f7920640 10 monclient: init 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.894+0000 7f36f7920640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.894+0000 7f36f7920640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.894+0000 7f36f7920640 10 monclient(hunting): picked mon.noname-c con 0x7f36f00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.894+0000 7f36f7920640 10 monclient(hunting): picked mon.noname-a con 0x7f36f013f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.894+0000 7f36f7920640 10 monclient(hunting): picked mon.noname-b con 0x7f36f01430c0 addr v1:172.21.15.159:6789/0 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.894+0000 7f36f7920640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f013c190 con 0x7f36f013f710 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.895+0000 7f36f7920640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f00b7cf0 con 0x7f36f01430c0 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.895+0000 7f36f7920640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f013ad50 con 0x7f36f00a98f0 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.895+0000 7f36f7920640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:55.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.895+0000 7f36f7920640 10 monclient(hunting): authenticate will time out at 6568.053686s 2024-03-25T19:38:55.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.901+0000 7f36f56bc640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f36f00a98f0 0x7f36f013ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:56046/0 (socket says 172.21.15.134:56046) 2024-03-25T19:38:55.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.901+0000 7f36f56bc640 1 -- 172.21.15.134:0/1460757731 learned_addr learned my addr 172.21.15.134:0/1460757731 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:38:55.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.901+0000 7f36e5ffb640 10 client.?.objecter ms_handle_connect 0x7f36f00a98f0 2024-03-25T19:38:55.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.901+0000 7f36e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:55.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.905+0000 7f36e5ffb640 10 client.?.objecter ms_handle_connect 0x7f36f013f710 2024-03-25T19:38:55.905 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.906+0000 7f36e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:55.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.908+0000 7f36e5ffb640 10 client.?.objecter ms_handle_connect 0x7f36f01430c0 2024-03-25T19:38:55.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.908+0000 7f36e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:55.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.911+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652350248 0 0) 0x7f36f013ad50 con 0x7f36f00a98f0 2024-03-25T19:38:55.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.911+0000 7f36e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:55.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.911+0000 7f36e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:55.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.911+0000 7f36e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:55.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.911+0000 7f36e5ffb640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:38:55.911 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.911+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d4004eb0 con 0x7f36f00a98f0 2024-03-25T19:38:55.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.916+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146182029 0 0) 0x7f36f00b28a0 con 0x7f36f013f710 2024-03-25T19:38:55.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.916+0000 7f36e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:55.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.916+0000 7f36e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:55.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.916+0000 7f36e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:55.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.916+0000 7f36e5ffb640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:38:55.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.916+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f013ad50 con 0x7f36f013f710 2024-03-25T19:38:55.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.929+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1171020726 0 0) 0x7f36d4004eb0 con 0x7f36f00a98f0 2024-03-25T19:38:55.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.929+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f00b28a0 con 0x7f36f00a98f0 2024-03-25T19:38:55.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.929+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f36ec001950 con 0x7f36f00a98f0 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.929+0000 7f36e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.929+0000 7f36e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.929+0000 7f36e5ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:55.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.929+0000 7f36e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:55.933 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.934+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318457173 0 0) 0x7f36f00b7cf0 con 0x7f36f01430c0 2024-03-25T19:38:55.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.934+0000 7f36e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:55.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.934+0000 7f36e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:55.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.934+0000 7f36e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:55.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.934+0000 7f36e5ffb640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:38:55.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.934+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d4004eb0 con 0x7f36f01430c0 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.936+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762742702 0 0) 0x7f36f013ad50 con 0x7f36f013f710 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.936+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f00b7cf0 con 0x7f36f013f710 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.936+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f36e00016d0 con 0x7f36f013f710 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.936+0000 7f36e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.936+0000 7f36e5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.936+0000 7f36e5ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:55.936 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.937 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:55.937 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:55.937 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:55.937 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:55.937 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:55.937 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:55.937 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:55.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4009845005 0 0) 0x7f36f00b28a0 con 0x7f36f00a98f0 2024-03-25T19:38:55.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 >> v1:172.21.15.159:6789/0 conn(0x7f36f01430c0 legacy=0x7f36f01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:55.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 >> v1:172.21.15.134:6789/0 conn(0x7f36f013f710 legacy=0x7f36f01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:55.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:55.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 1 monclient: found mon.c 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f013afc0 con 0x7f36f00a98f0 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 5 monclient: authenticate success, global_id 4691 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 10 monclient: _renew_subs 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f013b2a0 con 0x7f36f00a98f0 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 10 monclient: _renew_subs 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:55.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f012fa90 con 0x7f36f00a98f0 2024-03-25T19:38:55.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.941+0000 7f36f7920640 1 librados: init done 2024-03-25T19:38:55.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.942+0000 7f36f7920640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:55.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.942+0000 7f36f7920640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:55.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.942+0000 7f36f7920640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:55.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:55.942+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f00b3ef0 con 0x7f36f00a98f0 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36ec0047e0 con 0x7f36f00a98f0 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f36ec004af0 con 0x7f36f00a98f0 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e5ffb640 10 monclient: dump: 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:56.235 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e77fe640 4 set_mon_vals no callback set 2024-03-25T19:38:56.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.235+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f36ec01aac0 con 0x7f36f00a98f0 2024-03-25T19:38:56.237 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.236+0000 7f36f4ebb640 10 monclient: get_auth_request con 0x7f36d406df70 auth_method 0 2024-03-25T19:38:56.237 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.236+0000 7f36e5ffb640 10 client.4691.objecter ms_handle_connect 0x7f36d406df70 2024-03-25T19:38:56.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7f36ec054850 con 0x7f36f00a98f0 2024-03-25T19:38:56.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 10 client.4691.objecter ms_dispatch 0x7f36f0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:56.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:56.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:56.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-25T19:38:56.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36ec050190 con 0x7f36f00a98f0 2024-03-25T19:38:56.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:56.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.244+0000 7f36e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:56.464 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.464+0000 7f36f7920640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:56.464 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.464+0000 7f36f7920640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:56.464 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.464+0000 7f36f7920640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:38:56.464 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.464+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36f00627b0 con 0x7f36f00a98f0 2024-03-25T19:38:56.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.497+0000 7f36e5ffb640 1 -- 172.21.15.134:0/1460757731 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7f36ec052290 con 0x7f36f00a98f0 2024-03-25T19:38:56.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.497+0000 7f36e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:56.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.497+0000 7f36e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:56.497 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:38:56.498 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:56.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 10 librados: watch_flush enter 2024-03-25T19:38:56.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 10 librados: watch_flush exit 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 10 client.4691.objecter successfully canceled tick 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 >> v1:172.21.15.159:6800/1875874887 conn(0x7f36d406df70 legacy=0x7f36d4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 10 monclient: shutdown 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 >> v1:172.21.15.134:6790/0 conn(0x7f36f00a98f0 legacy=0x7f36f013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 shutdown_connections 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 >> 172.21.15.134:0/1460757731 conn(0x7f36f00b7ea0 msgr2=0x7f36f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 shutdown_connections 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 1 -- 172.21.15.134:0/1460757731 wait complete. 2024-03-25T19:38:56.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:38:56.500+0000 7f36f7920640 1 librados: shutdown 2024-03-25T19:38:56.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:03.320 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:03.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.428+0000 7f1691fef640 10 monclient: get_monmap_and_config 2024-03-25T19:39:03.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.428+0000 7f1691fef640 10 monclient: build_initial_monmap 2024-03-25T19:39:03.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.428+0000 7f1691fef640 10 monclient: monmap: 2024-03-25T19:39:03.428 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:03.428 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:03.428 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:39:03.429708+0000 2024-03-25T19:39:03.428 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:39:03.429708+0000 2024-03-25T19:39:03.429 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:03.429 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:03.429 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:03.429 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:39:03.429 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:39:03.429 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.429+0000 7f1691fef640 1 Processor -- start 2024-03-25T19:39:03.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.429+0000 7f1691fef640 1 -- start start 2024-03-25T19:39:03.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.429+0000 7f1691fef640 10 monclient: init 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.429+0000 7f1691fef640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.429+0000 7f1691fef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.429+0000 7f1691fef640 10 monclient(hunting): picked mon.noname-b con 0x7f168c070410 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.430+0000 7f1691fef640 10 monclient(hunting): picked mon.noname-c con 0x7f168c0707f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.430+0000 7f1691fef640 10 monclient(hunting): picked mon.noname-a con 0x7f168c13db40 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.430+0000 7f1691fef640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c0adea0 con 0x7f168c13db40 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.430+0000 7f1691fef640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c1445a0 con 0x7f168c0707f0 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.430+0000 7f1691fef640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c0720a0 con 0x7f168c070410 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.430+0000 7f1691fef640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:03.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.430+0000 7f1691fef640 10 monclient(hunting): authenticate will time out at 6575.588693s 2024-03-25T19:39:03.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.459+0000 7f168affd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f168c0707f0 0x7f168c13d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58670/0 (socket says 172.21.15.134:58670) 2024-03-25T19:39:03.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.459+0000 7f168affd640 1 -- 172.21.15.134:0/864367857 learned_addr learned my addr 172.21.15.134:0/864367857 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:03.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.479+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240414108 0 0) 0x7f168c1445a0 con 0x7f168c0707f0 2024-03-25T19:39:03.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.479+0000 7f168a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:03.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.479+0000 7f168a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:03.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.479+0000 7f168a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:03.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.479+0000 7f168a7fc640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:39:03.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.479+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1674003520 con 0x7f168c0707f0 2024-03-25T19:39:03.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.482+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185701965 0 0) 0x7f168c0720a0 con 0x7f168c070410 2024-03-25T19:39:03.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.482+0000 7f168a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:03.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.482+0000 7f168a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:03.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.482+0000 7f168a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:03.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.482+0000 7f168a7fc640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:39:03.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.482+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f168c1445a0 con 0x7f168c070410 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896949355 0 0) 0x7f1674003520 con 0x7f168c0707f0 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 >> v1:172.21.15.134:6790/0 conn(0x7f168c070410 legacy=0x7f168c143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 >> v1:172.21.15.134:6789/0 conn(0x7f168c13db40 legacy=0x7f168c14ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c072380 con 0x7f168c0707f0 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1680001690 con 0x7f168c0707f0 2024-03-25T19:39:03.487 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f1691fef640 5 monclient: authenticate success, global_id 4684 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f1691fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.486+0000 7f168a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.487+0000 7f168a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.487+0000 7f168a7fc640 10 monclient: dump: 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:03.488 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.487+0000 7f1691fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16800017f0 con 0x7f168c0707f0 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f168a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f168a7fc640 1 -- 172.21.15.134:0/864367857 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1680001c00 con 0x7f168c0707f0 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f1691fef640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f1691fef640 4 set_mon_vals no callback set 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f168a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f168a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.500+0000 7f168a7fc640 10 monclient: dump: 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.500 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:03.501 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:03.501 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:03.501 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:03.501 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:03.501 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.501+0000 7f1691fef640 10 monclient: shutdown 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.501+0000 7f1691fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.501+0000 7f1691fef640 1 -- 172.21.15.134:0/864367857 >> v1:172.21.15.159:6789/0 conn(0x7f168c0707f0 legacy=0x7f168c13d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.501+0000 7f1691fef640 1 -- 172.21.15.134:0/864367857 shutdown_connections 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.501+0000 7f1691fef640 1 -- 172.21.15.134:0/864367857 >> 172.21.15.134:0/864367857 conn(0x7f168c06b630 msgr2=0x7f168c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.501+0000 7f1691fef640 1 -- 172.21.15.134:0/864367857 shutdown_connections 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.501+0000 7f1691fef640 1 -- 172.21.15.134:0/864367857 wait complete. 2024-03-25T19:39:03.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.502+0000 7f1691fef640 10 monclient: build_initial_monmap 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.502+0000 7f1691fef640 10 monclient: monmap: 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.502+0000 7f1691fef640 1 librados: starting msgr at 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.502+0000 7f1691fef640 1 librados: starting objecter 2024-03-25T19:39:03.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.502+0000 7f1691fef640 1 Processor -- start 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 1 -- start start 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 1 librados: setting wanted keys 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 1 librados: calling monclient init 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 10 monclient: init 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 10 monclient(hunting): picked mon.noname-c con 0x7f168c070410 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 10 monclient(hunting): picked mon.noname-a con 0x7f168c0707f0 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 10 monclient(hunting): picked mon.noname-b con 0x7f168c13db40 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.503+0000 7f1691fef640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c0ab770 con 0x7f168c0707f0 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.504+0000 7f1691fef640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c0a9810 con 0x7f168c13db40 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.504+0000 7f1691fef640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c140ed0 con 0x7f168c070410 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.504+0000 7f1691fef640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:03.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.504+0000 7f1691fef640 10 monclient(hunting): authenticate will time out at 6575.662735s 2024-03-25T19:39:03.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.514+0000 7f168bfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f168c13db40 0x7f168c1e3250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58698/0 (socket says 172.21.15.134:58698) 2024-03-25T19:39:03.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.514+0000 7f168bfff640 1 -- 172.21.15.134:0/2060658687 learned_addr learned my addr 172.21.15.134:0/2060658687 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:03.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.514+0000 7f1673fff640 10 client.?.objecter ms_handle_connect 0x7f168c13db40 2024-03-25T19:39:03.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.514+0000 7f1673fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:03.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.515+0000 7f1673fff640 10 client.?.objecter ms_handle_connect 0x7f168c070410 2024-03-25T19:39:03.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.515+0000 7f1673fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:03.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.520+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764048125 0 0) 0x7f168c0a9810 con 0x7f168c13db40 2024-03-25T19:39:03.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.520+0000 7f1673fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:03.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.520+0000 7f1673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:03.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.520+0000 7f1673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:03.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.520+0000 7f1673fff640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:39:03.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.520+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678004eb0 con 0x7f168c13db40 2024-03-25T19:39:03.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.523+0000 7f1673fff640 10 client.?.objecter ms_handle_connect 0x7f168c0707f0 2024-03-25T19:39:03.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.523+0000 7f1673fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:03.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.534+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042914457 0 0) 0x7f168c0ab770 con 0x7f168c0707f0 2024-03-25T19:39:03.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.535+0000 7f1673fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:03.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.535+0000 7f1673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:03.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.535+0000 7f1673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:03.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.535+0000 7f1673fff640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:39:03.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.535+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f168c0a9810 con 0x7f168c0707f0 2024-03-25T19:39:03.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.536+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021454152 0 0) 0x7f168c140ed0 con 0x7f168c070410 2024-03-25T19:39:03.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.536+0000 7f1673fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:03.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.536+0000 7f1673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:03.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.536+0000 7f1673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:03.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.536+0000 7f1673fff640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:39:03.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.536+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f168c0ab770 con 0x7f168c070410 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.537+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042779083 0 0) 0x7f1678004eb0 con 0x7f168c13db40 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.538+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f168c140ed0 con 0x7f168c13db40 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.538+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1684001c60 con 0x7f168c13db40 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.538+0000 7f1673fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.538+0000 7f1673fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.538+0000 7f1673fff640 10 monclient(hunting): dump: 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:03.538 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:03.539 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:03.539 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.538+0000 7f1673fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:03.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.547+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3101596935 0 0) 0x7f168c0ab770 con 0x7f168c070410 2024-03-25T19:39:03.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.547+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1678004eb0 con 0x7f168c070410 2024-03-25T19:39:03.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f167c001930 con 0x7f168c070410 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 10 monclient(hunting): dump: 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:03.548 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 668435194 0 0) 0x7f168c0a9810 con 0x7f168c0707f0 2024-03-25T19:39:03.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f168c0ab770 con 0x7f168c0707f0 2024-03-25T19:39:03.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1680001b00 con 0x7f168c0707f0 2024-03-25T19:39:03.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.548+0000 7f1673fff640 10 monclient(hunting): dump: 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:03.551 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2759727240 0 0) 0x7f168c140ed0 con 0x7f168c13db40 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 >> v1:172.21.15.134:6790/0 conn(0x7f168c070410 legacy=0x7f168c1381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 >> v1:172.21.15.134:6789/0 conn(0x7f168c0707f0 legacy=0x7f168c1db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 1 monclient: found mon.b 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c141060 con 0x7f168c13db40 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 10 monclient: _finish_auth 0 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 5 monclient: authenticate success, global_id 4687 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 10 monclient: _renew_subs 2024-03-25T19:39:03.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c138900 con 0x7f168c13db40 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 10 monclient: _renew_subs 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c13e7e0 con 0x7f168c13db40 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.549+0000 7f1691fef640 1 librados: init done 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.550+0000 7f1691fef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.550+0000 7f1691fef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.550+0000 7f1691fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:03.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.550+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c138c50 con 0x7f168c13db40 2024-03-25T19:39:03.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.766+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1684001dc0 con 0x7f168c13db40 2024-03-25T19:39:03.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.766+0000 7f1673fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:03.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.766+0000 7f1689ffb640 4 set_mon_vals no callback set 2024-03-25T19:39:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.827+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1684003b30 con 0x7f168c13db40 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.827+0000 7f1673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.827+0000 7f1673fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.827+0000 7f1673fff640 10 monclient: dump: 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:03.827 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:03.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.828+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f1684003d50 con 0x7f168c13db40 2024-03-25T19:39:03.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.828+0000 7f168b7fe640 10 monclient: get_auth_request con 0x7f167806e060 auth_method 0 2024-03-25T19:39:03.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.829+0000 7f1673fff640 10 client.4687.objecter ms_handle_connect 0x7f167806e060 2024-03-25T19:39:03.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.831+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7f1684054b90 con 0x7f168c13db40 2024-03-25T19:39:03.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.831+0000 7f1673fff640 10 client.4687.objecter ms_dispatch 0x7f168c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:03.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.831+0000 7f1673fff640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:03.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.831+0000 7f1673fff640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:03.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.831+0000 7f1673fff640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T19:39:03.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.835+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16840504d0 con 0x7f168c13db40 2024-03-25T19:39:03.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.835+0000 7f1673fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:03.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:03.835+0000 7f1673fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:04.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.082+0000 7f1691fef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:04.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.082+0000 7f1691fef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:04.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.082+0000 7f1691fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:04.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.082+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f168c0627b0 con 0x7f168c13db40 2024-03-25T19:39:04.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.115+0000 7f1673fff640 1 -- 172.21.15.134:0/2060658687 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7f16840525d0 con 0x7f168c13db40 2024-03-25T19:39:04.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.115+0000 7f1673fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:04.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.115+0000 7f1673fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:04.116 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:39:04.116 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:04.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 10 librados: watch_flush enter 2024-03-25T19:39:04.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 10 librados: watch_flush exit 2024-03-25T19:39:04.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 10 client.4687.objecter successfully canceled tick 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 >> v1:172.21.15.159:6800/1875874887 conn(0x7f167806e060 legacy=0x7f1678070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 10 monclient: shutdown 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 >> v1:172.21.15.159:6789/0 conn(0x7f168c13db40 legacy=0x7f168c1e3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 shutdown_connections 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 >> 172.21.15.134:0/2060658687 conn(0x7f168c06b630 msgr2=0x7f168c070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 shutdown_connections 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.118+0000 7f1691fef640 1 -- 172.21.15.134:0/2060658687 wait complete. 2024-03-25T19:39:04.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:04.119+0000 7f1691fef640 1 librados: shutdown 2024-03-25T19:39:04.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:10.936 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.058+0000 7f1192686640 10 monclient: get_monmap_and_config 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.058+0000 7f1192686640 10 monclient: build_initial_monmap 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.058+0000 7f1192686640 10 monclient: monmap: 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:39:11.059182+0000 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:39:11.059182+0000 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:39:11.058 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:39:11.059 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 1 Processor -- start 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 1 -- start start 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient: init 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient(hunting): picked mon.noname-b con 0x7f118c0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient(hunting): picked mon.noname-c con 0x7f118c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient(hunting): picked mon.noname-a con 0x7f118c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0af940 con 0x7f118c14be90 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0b0a70 con 0x7f118c0b1730 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0b0d50 con 0x7f118c0b4d60 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:11.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.059+0000 7f1192686640 10 monclient(hunting): authenticate will time out at 6583.218494s 2024-03-25T19:39:11.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.062+0000 7f118bfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f118c0b4d60 0x7f118c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59238/0 (socket says 172.21.15.134:59238) 2024-03-25T19:39:11.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.062+0000 7f118bfff640 1 -- 172.21.15.134:0/3880223918 learned_addr learned my addr 172.21.15.134:0/3880223918 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:11.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.072+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014338152 0 0) 0x7f118c0b0d50 con 0x7f118c0b4d60 2024-03-25T19:39:11.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.072+0000 7f118affd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:11.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.072+0000 7f118affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:11.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.072+0000 7f118affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:11.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.072+0000 7f118affd640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:39:11.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.072+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003520 con 0x7f118c0b4d60 2024-03-25T19:39:11.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.076+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614271064 0 0) 0x7f118c0b0a70 con 0x7f118c0b1730 2024-03-25T19:39:11.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.076+0000 7f118affd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:11.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.076+0000 7f118affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:11.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.076+0000 7f118affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:11.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.076+0000 7f118affd640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:39:11.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.076+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003840 con 0x7f118c0b1730 2024-03-25T19:39:11.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026440407 0 0) 0x7f1174003840 con 0x7f118c0b1730 2024-03-25T19:39:11.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 >> v1:172.21.15.134:6790/0 conn(0x7f118c0b4d60 legacy=0x7f118c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 >> v1:172.21.15.134:6789/0 conn(0x7f118c14be90 legacy=0x7f118c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 1 monclient: found mon.noname-c 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0b1030 con 0x7f118c0b1730 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 10 monclient: _finish_auth 0 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1180001670 con 0x7f118c0b1730 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f118affd640 10 monclient: dump: 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.081 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:11.082 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:11.082 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:11.082 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:11.082 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:11.082 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:11.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f1192686640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:39:11.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.080+0000 7f1192686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11800017d0 con 0x7f118c0b1730 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f118affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f118affd640 1 -- 172.21.15.134:0/3880223918 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1180001ba0 con 0x7f118c0b1730 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f1192686640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f1192686640 4 set_mon_vals no callback set 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f118affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f118affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.088+0000 7f118affd640 10 monclient: dump: 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:11.088 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:11.089 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:11.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.089+0000 7f1192686640 10 monclient: shutdown 2024-03-25T19:39:11.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.089+0000 7f1192686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.089+0000 7f1192686640 1 -- 172.21.15.134:0/3880223918 >> v1:172.21.15.159:6789/0 conn(0x7f118c0b1730 legacy=0x7f118c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.089+0000 7f1192686640 1 -- 172.21.15.134:0/3880223918 shutdown_connections 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.089+0000 7f1192686640 1 -- 172.21.15.134:0/3880223918 >> 172.21.15.134:0/3880223918 conn(0x7f118c0617a0 msgr2=0x7f118c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.089+0000 7f1192686640 1 -- 172.21.15.134:0/3880223918 shutdown_connections 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.089+0000 7f1192686640 1 -- 172.21.15.134:0/3880223918 wait complete. 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 10 monclient: build_initial_monmap 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 10 monclient: monmap: 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:11.091 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 1 librados: starting msgr at 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 1 librados: starting objecter 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 1 Processor -- start 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 1 -- start start 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 1 librados: setting wanted keys 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 1 librados: calling monclient init 2024-03-25T19:39:11.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.090+0000 7f1192686640 10 monclient: init 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 10 monclient(hunting): picked mon.noname-b con 0x7f118c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 10 monclient(hunting): picked mon.noname-a con 0x7f118c0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 10 monclient(hunting): picked mon.noname-c con 0x7f118c14be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c13ed80 con 0x7f118c0b4d60 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c13ea10 con 0x7f118c0b1730 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c137c00 con 0x7f118c14be90 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:11.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.091+0000 7f1192686640 10 monclient(hunting): authenticate will time out at 6583.250451s 2024-03-25T19:39:11.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.098+0000 7f1190c23640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f118c14be90 0x7f118c1db340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59254/0 (socket says 172.21.15.134:59254) 2024-03-25T19:39:11.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.098+0000 7f1190c23640 1 -- 172.21.15.134:0/3087234620 learned_addr learned my addr 172.21.15.134:0/3087234620 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:11.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.099+0000 7f1188ff9640 10 client.?.objecter ms_handle_connect 0x7f118c14be90 2024-03-25T19:39:11.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.099+0000 7f1188ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:11.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.102+0000 7f1188ff9640 10 client.?.objecter ms_handle_connect 0x7f118c0b4d60 2024-03-25T19:39:11.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.102+0000 7f1188ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:11.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.110+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289541404 0 0) 0x7f118c137c00 con 0x7f118c14be90 2024-03-25T19:39:11.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.110+0000 7f1188ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:11.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.110+0000 7f1188ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:11.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.110+0000 7f1188ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:11.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.110+0000 7f1188ff9640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:39:11.111 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.110+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1178004eb0 con 0x7f118c14be90 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560230849 0 0) 0x7f118c13ed80 con 0x7f118c0b4d60 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c137c00 con 0x7f118c0b4d60 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 82979585 0 0) 0x7f1178004eb0 con 0x7f118c14be90 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f118c13ed80 con 0x7f118c14be90 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1184001c30 con 0x7f118c14be90 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:11.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:11.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.121+0000 7f1188ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:11.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.128+0000 7f1188ff9640 10 client.?.objecter ms_handle_connect 0x7f118c0b1730 2024-03-25T19:39:11.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.128+0000 7f1188ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:11.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 316936659 0 0) 0x7f118c13ed80 con 0x7f118c14be90 2024-03-25T19:39:11.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 >> v1:172.21.15.159:6789/0 conn(0x7f118c0b1730 legacy=0x7f118c134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 >> v1:172.21.15.134:6789/0 conn(0x7f118c0b4d60 legacy=0x7f118c1dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 1 monclient: found mon.c 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c137d90 con 0x7f118c14be90 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1188ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 5 monclient: authenticate success, global_id 4703 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 10 monclient: _renew_subs 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c135550 con 0x7f118c14be90 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:11.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 10 monclient: _renew_subs 2024-03-25T19:39:11.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:11.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c1407d0 con 0x7f118c14be90 2024-03-25T19:39:11.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.136+0000 7f1192686640 1 librados: init done 2024-03-25T19:39:11.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.138+0000 7f1192686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:11.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.138+0000 7f1192686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:11.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.138+0000 7f1192686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:11.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.138+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c135850 con 0x7f118c14be90 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.436+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1184003970 con 0x7f118c14be90 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.436+0000 7f1188ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.436+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f1184003d80 con 0x7f118c14be90 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.436+0000 7f1188ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.436+0000 7f1188ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.436+0000 7f1188ff9640 10 monclient: dump: 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:11.436 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.436+0000 7f118a7fc640 4 set_mon_vals no callback set 2024-03-25T19:39:11.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.437+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f1184003fa0 con 0x7f118c14be90 2024-03-25T19:39:11.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.437+0000 7f118bfff640 10 monclient: get_auth_request con 0x7f117806dda0 auth_method 0 2024-03-25T19:39:11.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.438+0000 7f1188ff9640 10 client.4703.objecter ms_handle_connect 0x7f117806dda0 2024-03-25T19:39:11.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7f11840549e0 con 0x7f118c14be90 2024-03-25T19:39:11.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 10 client.4703.objecter ms_dispatch 0x7f118c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:11.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:11.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:11.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T19:39:11.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11840504f0 con 0x7f118c14be90 2024-03-25T19:39:11.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:11.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.445+0000 7f1188ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:11.641 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:16.314 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.653+0000 7f1192686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:11.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.653+0000 7f1192686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:11.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.653+0000 7f1192686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:11.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.653+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f118c13eba0 con 0x7f118c14be90 2024-03-25T19:39:11.670 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:16.343 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.700 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:16.373 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.711 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:16.383 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.716 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:16.389 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.727 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:16.374 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.733 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:16.380 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.738+0000 7f1188ff9640 1 -- 172.21.15.134:0/3087234620 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7f1184052420 con 0x7f118c14be90 2024-03-25T19:39:11.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.738+0000 7f1188ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:11.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.738+0000 7f1188ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:11.739 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:39:11.739 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:11.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 10 librados: watch_flush enter 2024-03-25T19:39:11.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 10 librados: watch_flush exit 2024-03-25T19:39:11.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T19:39:11.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 10 client.4703.objecter successfully canceled tick 2024-03-25T19:39:11.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 >> v1:172.21.15.159:6800/1875874887 conn(0x7f117806dda0 legacy=0x7f1178070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 10 monclient: shutdown 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 >> v1:172.21.15.134:6790/0 conn(0x7f118c14be90 legacy=0x7f118c1db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 shutdown_connections 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.741+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 >> 172.21.15.134:0/3087234620 conn(0x7f118c0617a0 msgr2=0x7f118c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.742+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 shutdown_connections 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.742+0000 7f1192686640 1 -- 172.21.15.134:0/3087234620 wait complete. 2024-03-25T19:39:11.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:11.742+0000 7f1192686640 1 librados: shutdown 2024-03-25T19:39:11.754 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:11.767 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:16.414 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:11.770 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:16.442 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.541 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:22.214 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.550 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:22.222 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.560 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:22.207 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.560 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:22.233 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.564 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:22.211 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.565 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:22.237 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.572 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:22.245 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.582 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:22.229 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:17.630 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:22.302 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.560 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:18.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.661+0000 7efe54d7b640 10 monclient: get_monmap_and_config 2024-03-25T19:39:18.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.661+0000 7efe54d7b640 10 monclient: build_initial_monmap 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.661+0000 7efe54d7b640 10 monclient: monmap: 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:39:18.662970+0000 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:39:18.662970+0000 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:39:18.662 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.662+0000 7efe54d7b640 1 Processor -- start 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.662+0000 7efe54d7b640 1 -- start start 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.662+0000 7efe54d7b640 10 monclient: init 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 10 monclient(hunting): picked mon.noname-c con 0x7efe500ae6e0 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 10 monclient(hunting): picked mon.noname-b con 0x7efe5006f080 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 10 monclient(hunting): picked mon.noname-a con 0x7efe500704e0 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500718d0 con 0x7efe500704e0 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5006fa30 con 0x7efe500ae6e0 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe50070090 con 0x7efe5006f080 2024-03-25T19:39:18.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:18.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe54d7b640 10 monclient(hunting): authenticate will time out at 6590.822050s 2024-03-25T19:39:18.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe4e59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7efe500ae6e0 0x7efe5013c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52056/0 (socket says 172.21.15.134:52056) 2024-03-25T19:39:18.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.663+0000 7efe4e59c640 1 -- 172.21.15.134:0/576561275 learned_addr learned my addr 172.21.15.134:0/576561275 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:18.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.691+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584971088 0 0) 0x7efe5006fa30 con 0x7efe500ae6e0 2024-03-25T19:39:18.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.691+0000 7efe4dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:18.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.691+0000 7efe4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:18.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.691+0000 7efe4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:18.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.691+0000 7efe4dd9b640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:39:18.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.691+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003520 con 0x7efe500ae6e0 2024-03-25T19:39:18.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.693+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463950027 0 0) 0x7efe500718d0 con 0x7efe500704e0 2024-03-25T19:39:18.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.693+0000 7efe4dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:18.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.693+0000 7efe4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:18.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.693+0000 7efe4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:18.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.693+0000 7efe4dd9b640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:39:18.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.693+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5006fa30 con 0x7efe500704e0 2024-03-25T19:39:18.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.694+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378349305 0 0) 0x7efe50070090 con 0x7efe5006f080 2024-03-25T19:39:18.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.694+0000 7efe4dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:18.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.694+0000 7efe4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:18.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.694+0000 7efe4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:18.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.694+0000 7efe4dd9b640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:39:18.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.694+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe500718d0 con 0x7efe5006f080 2024-03-25T19:39:18.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642406965 0 0) 0x7efe5006fa30 con 0x7efe500704e0 2024-03-25T19:39:18.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 >> v1:172.21.15.134:6790/0 conn(0x7efe5006f080 legacy=0x7efe50071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 >> v1:172.21.15.159:6789/0 conn(0x7efe500ae6e0 legacy=0x7efe5013c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5006f510 con 0x7efe500704e0 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe54d7b640 5 monclient: authenticate success, global_id 4734 2024-03-25T19:39:18.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe54d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe480019b0 con 0x7efe500704e0 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe4dd9b640 10 monclient: dump: 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:18.709 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:18.710 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:18.710 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:18.710 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:18.710 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:18.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.708+0000 7efe54d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe48001c00 con 0x7efe500704e0 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe4dd9b640 1 -- 172.21.15.134:0/576561275 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe48001f90 con 0x7efe500704e0 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe54d7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe54d7b640 4 set_mon_vals no callback set 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe4dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.792+0000 7efe4dd9b640 10 monclient: dump: 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:18.792 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:18.793 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:18.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.793+0000 7efe54d7b640 10 monclient: shutdown 2024-03-25T19:39:18.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.793+0000 7efe54d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.793+0000 7efe54d7b640 1 -- 172.21.15.134:0/576561275 >> v1:172.21.15.134:6789/0 conn(0x7efe500704e0 legacy=0x7efe5014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.793+0000 7efe54d7b640 1 -- 172.21.15.134:0/576561275 shutdown_connections 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.793+0000 7efe54d7b640 1 -- 172.21.15.134:0/576561275 >> 172.21.15.134:0/576561275 conn(0x7efe5006b630 msgr2=0x7efe500b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.793+0000 7efe54d7b640 1 -- 172.21.15.134:0/576561275 shutdown_connections 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.793+0000 7efe54d7b640 1 -- 172.21.15.134:0/576561275 wait complete. 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 10 monclient: build_initial_monmap 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 10 monclient: monmap: 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:18.795 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 1 librados: starting msgr at 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 1 librados: starting objecter 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 1 Processor -- start 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 1 -- start start 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 1 librados: setting wanted keys 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 1 librados: calling monclient init 2024-03-25T19:39:18.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.794+0000 7efe54d7b640 10 monclient: init 2024-03-25T19:39:18.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:18.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:18.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 10 monclient(hunting): picked mon.noname-b con 0x7efe500704e0 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:18.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 10 monclient(hunting): picked mon.noname-a con 0x7efe500ae6e0 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:18.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 10 monclient(hunting): picked mon.noname-c con 0x7efe501d5fd0 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:18.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500ab5b0 con 0x7efe500ae6e0 2024-03-25T19:39:18.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe501d5ab0 con 0x7efe500704e0 2024-03-25T19:39:18.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500ad710 con 0x7efe501d5fd0 2024-03-25T19:39:18.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:18.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.795+0000 7efe54d7b640 10 monclient(hunting): authenticate will time out at 6590.954465s 2024-03-25T19:39:18.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.802+0000 7efe4e59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7efe500704e0 0x7efe501d5290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52084/0 (socket says 172.21.15.134:52084) 2024-03-25T19:39:18.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.802+0000 7efe4e59c640 1 -- 172.21.15.134:0/2445307354 learned_addr learned my addr 172.21.15.134:0/2445307354 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:18.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.803+0000 7efe46ffd640 10 client.?.objecter ms_handle_connect 0x7efe500704e0 2024-03-25T19:39:18.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.803+0000 7efe46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:18.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.812+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996750065 0 0) 0x7efe501d5ab0 con 0x7efe500704e0 2024-03-25T19:39:18.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.812+0000 7efe46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:18.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.812+0000 7efe46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:18.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.812+0000 7efe46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:18.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.812+0000 7efe46ffd640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:39:18.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.812+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe30004eb0 con 0x7efe500704e0 2024-03-25T19:39:18.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.813+0000 7efe46ffd640 10 client.?.objecter ms_handle_connect 0x7efe500ae6e0 2024-03-25T19:39:18.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.813+0000 7efe46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:18.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.814+0000 7efe46ffd640 10 client.?.objecter ms_handle_connect 0x7efe501d5fd0 2024-03-25T19:39:18.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.814+0000 7efe46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.819+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1288454075 0 0) 0x7efe30004eb0 con 0x7efe500704e0 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.819+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe501d5ab0 con 0x7efe500704e0 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.819+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe40001830 con 0x7efe500704e0 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.819+0000 7efe46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.819+0000 7efe46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.819+0000 7efe46ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.819 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.820 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:18.820 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:18.820 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:18.820 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:18.820 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:18.820 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:18.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.819+0000 7efe46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:18.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.822+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983971082 0 0) 0x7efe500ad710 con 0x7efe501d5fd0 2024-03-25T19:39:18.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.822+0000 7efe46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:18.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.822+0000 7efe46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:18.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.822+0000 7efe46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:18.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.822+0000 7efe46ffd640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:39:18.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.822+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe30004eb0 con 0x7efe501d5fd0 2024-03-25T19:39:18.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.823+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115005037 0 0) 0x7efe500ab5b0 con 0x7efe500ae6e0 2024-03-25T19:39:18.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.823+0000 7efe46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:18.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.823+0000 7efe46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:18.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.823+0000 7efe46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:18.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.823+0000 7efe46ffd640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:39:18.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.823+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe500ad710 con 0x7efe500ae6e0 2024-03-25T19:39:18.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.830+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1584937529 0 0) 0x7efe30004eb0 con 0x7efe501d5fd0 2024-03-25T19:39:18.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.830+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe500ab5b0 con 0x7efe501d5fd0 2024-03-25T19:39:18.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.833+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe48001f10 con 0x7efe501d5fd0 2024-03-25T19:39:18.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.833+0000 7efe46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:18.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.833+0000 7efe46ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:18.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.833+0000 7efe46ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:18.833 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.835+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106898650 0 0) 0x7efe500ad710 con 0x7efe500ae6e0 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.835+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe30004eb0 con 0x7efe500ae6e0 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.835+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe38001830 con 0x7efe500ae6e0 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.835+0000 7efe46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.835+0000 7efe46ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.835+0000 7efe46ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:18.835 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.836 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:18.836 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:18.836 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:18.836 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:18.836 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:18.836 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:18.836 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 125433957 0 0) 0x7efe501d5ab0 con 0x7efe500704e0 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 >> v1:172.21.15.134:6790/0 conn(0x7efe501d5fd0 legacy=0x7efe501dfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 >> v1:172.21.15.134:6789/0 conn(0x7efe500ae6e0 legacy=0x7efe500a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 1 monclient: found mon.b 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe500ad8a0 con 0x7efe500704e0 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe54d7b640 5 monclient: authenticate success, global_id 4699 2024-03-25T19:39:18.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe54d7b640 10 monclient: _renew_subs 2024-03-25T19:39:18.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe54d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:18.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.837+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe500accf0 con 0x7efe500704e0 2024-03-25T19:39:18.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:18.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 10 monclient: _renew_subs 2024-03-25T19:39:18.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:18.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe501e12c0 con 0x7efe500704e0 2024-03-25T19:39:18.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 1 librados: init done 2024-03-25T19:39:18.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:18.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:18.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:18.840 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:18.838+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe500627b0 con 0x7efe500704e0 2024-03-25T19:39:19.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.054+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe40001d20 con 0x7efe500704e0 2024-03-25T19:39:19.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.054+0000 7efe46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:19.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.054+0000 7efe4d59a640 4 set_mon_vals no callback set 2024-03-25T19:39:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.118+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe400049c0 con 0x7efe500704e0 2024-03-25T19:39:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.118+0000 7efe46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.118+0000 7efe46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.118+0000 7efe46ffd640 10 monclient: dump: 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:19.118 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:19.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.119+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7efe40004be0 con 0x7efe500704e0 2024-03-25T19:39:19.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.119+0000 7efe47fff640 10 monclient: get_auth_request con 0x7efe3006e0e0 auth_method 0 2024-03-25T19:39:19.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.119+0000 7efe46ffd640 10 client.4699.objecter ms_handle_connect 0x7efe3006e0e0 2024-03-25T19:39:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.121+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7efe40054860 con 0x7efe500704e0 2024-03-25T19:39:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.121+0000 7efe46ffd640 10 client.4699.objecter ms_dispatch 0x7efe50000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.121+0000 7efe46ffd640 3 client.4699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.121+0000 7efe46ffd640 3 client.4699.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.121+0000 7efe46ffd640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T19:39:19.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.123+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe400501a0 con 0x7efe500704e0 2024-03-25T19:39:19.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.123+0000 7efe46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:19.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.123+0000 7efe46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:19.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.375+0000 7efe54d7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:19.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.375+0000 7efe54d7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:19.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.375+0000 7efe54d7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:19.375 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.375+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe5006eb30 con 0x7efe500704e0 2024-03-25T19:39:19.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.406+0000 7efe46ffd640 1 -- 172.21.15.134:0/2445307354 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7efe400522a0 con 0x7efe500704e0 2024-03-25T19:39:19.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.406+0000 7efe46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:19.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.406+0000 7efe46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:19.407 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:39:19.407 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:19.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 10 librados: watch_flush enter 2024-03-25T19:39:19.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 10 librados: watch_flush exit 2024-03-25T19:39:19.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 10 client.4699.objecter successfully canceled tick 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 >> v1:172.21.15.159:6800/1875874887 conn(0x7efe3006e0e0 legacy=0x7efe300705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 10 monclient: shutdown 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 >> v1:172.21.15.159:6789/0 conn(0x7efe500704e0 legacy=0x7efe501d5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 shutdown_connections 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 >> 172.21.15.134:0/2445307354 conn(0x7efe5006b630 msgr2=0x7efe50071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 shutdown_connections 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 1 -- 172.21.15.134:0/2445307354 wait complete. 2024-03-25T19:39:19.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:19.409+0000 7efe54d7b640 1 librados: shutdown 2024-03-25T19:39:19.420 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:26.227 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:26.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.351+0000 7efeb15e4640 10 monclient: get_monmap_and_config 2024-03-25T19:39:26.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.351+0000 7efeb15e4640 10 monclient: build_initial_monmap 2024-03-25T19:39:26.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.351+0000 7efeb15e4640 10 monclient: monmap: 2024-03-25T19:39:26.351 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:26.351 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:26.351 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:39:26.352641+0000 2024-03-25T19:39:26.351 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:39:26.352641+0000 2024-03-25T19:39:26.352 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:26.352 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:26.352 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:26.352 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:39:26.352 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:39:26.352 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:26.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.352+0000 7efeb15e4640 1 Processor -- start 2024-03-25T19:39:26.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.352+0000 7efeb15e4640 1 -- start start 2024-03-25T19:39:26.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.352+0000 7efeb15e4640 10 monclient: init 2024-03-25T19:39:26.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:26.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:26.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 10 monclient(hunting): picked mon.noname-c con 0x7efeac0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:26.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 10 monclient(hunting): picked mon.noname-a con 0x7efeac13f710 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:26.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 10 monclient(hunting): picked mon.noname-b con 0x7efeac1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:26.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac142520 con 0x7efeac13f710 2024-03-25T19:39:26.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac0b28a0 con 0x7efeac0a98f0 2024-03-25T19:39:26.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac0b2a30 con 0x7efeac1430c0 2024-03-25T19:39:26.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:26.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.353+0000 7efeb15e4640 10 monclient(hunting): authenticate will time out at 6598.511835s 2024-03-25T19:39:26.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.363+0000 7efeab7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7efeac1430c0 0x7efeac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:41282/0 (socket says 172.21.15.134:41282) 2024-03-25T19:39:26.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.363+0000 7efeab7fe640 1 -- 172.21.15.134:0/4268191552 learned_addr learned my addr 172.21.15.134:0/4268191552 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:26.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.388+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610781677 0 0) 0x7efeac142520 con 0x7efeac13f710 2024-03-25T19:39:26.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.388+0000 7efeaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:26.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.388+0000 7efeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:26.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.388+0000 7efeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:26.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.388+0000 7efeaa7fc640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:39:26.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.388+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe98003520 con 0x7efeac13f710 2024-03-25T19:39:26.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.389+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585554855 0 0) 0x7efeac0b2a30 con 0x7efeac1430c0 2024-03-25T19:39:26.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.389+0000 7efeaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:26.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.389+0000 7efeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:26.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.389+0000 7efeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:26.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.389+0000 7efeaa7fc640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:39:26.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.389+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac142520 con 0x7efeac1430c0 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387155440 0 0) 0x7efe98003520 con 0x7efeac13f710 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 >> v1:172.21.15.134:6790/0 conn(0x7efeac1430c0 legacy=0x7efeac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 >> v1:172.21.15.159:6789/0 conn(0x7efeac0a98f0 legacy=0x7efeac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 1 monclient: found mon.noname-a 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac0b44c0 con 0x7efeac13f710 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:26.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe94001690 con 0x7efeac13f710 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeaa7fc640 10 monclient: dump: 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:26.394 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:26.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeb15e4640 5 monclient: authenticate success, global_id 4740 2024-03-25T19:39:26.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.393+0000 7efeb15e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:26.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe940017f0 con 0x7efeac13f710 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeaa7fc640 1 -- 172.21.15.134:0/4268191552 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efe94001c00 con 0x7efeac13f710 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeb15e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeb15e4640 4 set_mon_vals no callback set 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.413+0000 7efeaa7fc640 10 monclient: dump: 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:26.413 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:26.414 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:26.414 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:26.414 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:26.414 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:26.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.414+0000 7efeb15e4640 10 monclient: shutdown 2024-03-25T19:39:26.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.414+0000 7efeb15e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:26.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.414+0000 7efeb15e4640 1 -- 172.21.15.134:0/4268191552 >> v1:172.21.15.134:6789/0 conn(0x7efeac13f710 legacy=0x7efeac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:26.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.414+0000 7efeb15e4640 1 -- 172.21.15.134:0/4268191552 shutdown_connections 2024-03-25T19:39:26.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.414+0000 7efeb15e4640 1 -- 172.21.15.134:0/4268191552 >> 172.21.15.134:0/4268191552 conn(0x7efeac0b7ea0 msgr2=0x7efeac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.414+0000 7efeb15e4640 1 -- 172.21.15.134:0/4268191552 shutdown_connections 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.414+0000 7efeb15e4640 1 -- 172.21.15.134:0/4268191552 wait complete. 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.415+0000 7efeb15e4640 10 monclient: build_initial_monmap 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.415+0000 7efeb15e4640 10 monclient: monmap: 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:26.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.415+0000 7efeb15e4640 1 librados: starting msgr at 2024-03-25T19:39:26.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.415+0000 7efeb15e4640 1 librados: starting objecter 2024-03-25T19:39:26.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 1 Processor -- start 2024-03-25T19:39:26.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 1 -- start start 2024-03-25T19:39:26.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 1 librados: setting wanted keys 2024-03-25T19:39:26.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 1 librados: calling monclient init 2024-03-25T19:39:26.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient: init 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient(hunting): picked mon.noname-b con 0x7efeac0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient(hunting): picked mon.noname-c con 0x7efeac13f710 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient(hunting): picked mon.noname-a con 0x7efeac1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac137f10 con 0x7efeac1430c0 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac1dc070 con 0x7efeac0a98f0 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac137100 con 0x7efeac13f710 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:26.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.416+0000 7efeb15e4640 10 monclient(hunting): authenticate will time out at 6598.575612s 2024-03-25T19:39:26.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.430+0000 7efeaaffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7efeac0a98f0 0x7efeac137800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:46352/0 (socket says 172.21.15.134:46352) 2024-03-25T19:39:26.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.430+0000 7efeaaffd640 1 -- 172.21.15.134:0/3376408985 learned_addr learned my addr 172.21.15.134:0/3376408985 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:26.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.432+0000 7efea37fe640 10 client.?.objecter ms_handle_connect 0x7efeac0a98f0 2024-03-25T19:39:26.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.432+0000 7efea37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:26.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.440+0000 7efea37fe640 10 client.?.objecter ms_handle_connect 0x7efeac13f710 2024-03-25T19:39:26.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.440+0000 7efea37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:26.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.442+0000 7efea37fe640 10 client.?.objecter ms_handle_connect 0x7efeac1430c0 2024-03-25T19:39:26.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.442+0000 7efea37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:26.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.454+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961161511 0 0) 0x7efeac137f10 con 0x7efeac1430c0 2024-03-25T19:39:26.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.454+0000 7efea37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:26.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.454+0000 7efea37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:26.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.454+0000 7efea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:26.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.454+0000 7efea37fe640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:39:26.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.454+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe8c004eb0 con 0x7efeac1430c0 2024-03-25T19:39:26.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.456+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079979260 0 0) 0x7efeac137100 con 0x7efeac13f710 2024-03-25T19:39:26.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.456+0000 7efea37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:26.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.456+0000 7efea37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:26.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.456+0000 7efea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:26.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.456+0000 7efea37fe640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:39:26.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.456+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac137f10 con 0x7efeac13f710 2024-03-25T19:39:26.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.478+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2478850831 0 0) 0x7efe8c004eb0 con 0x7efeac1430c0 2024-03-25T19:39:26.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.478+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeac137100 con 0x7efeac1430c0 2024-03-25T19:39:26.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.478+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efea4001bd0 con 0x7efeac1430c0 2024-03-25T19:39:26.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.478+0000 7efea37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.478+0000 7efea37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.478+0000 7efea37fe640 10 monclient(hunting): dump: 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:26.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.478+0000 7efea37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780151598 0 0) 0x7efeac137100 con 0x7efeac1430c0 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 >> v1:172.21.15.134:6790/0 conn(0x7efeac13f710 legacy=0x7efeac1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 >> v1:172.21.15.159:6789/0 conn(0x7efeac0a98f0 legacy=0x7efeac137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 1 monclient: found mon.a 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac1373e0 con 0x7efeac1430c0 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:26.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efea37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 5 monclient: authenticate success, global_id 4743 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 10 monclient: _renew_subs 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac1358e0 con 0x7efeac1430c0 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 10 client.4743.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 10 monclient: _renew_subs 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac136150 con 0x7efeac1430c0 2024-03-25T19:39:26.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.561+0000 7efeb15e4640 1 librados: init done 2024-03-25T19:39:26.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.562+0000 7efeb15e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:26.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.562+0000 7efeb15e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:26.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.562+0000 7efeb15e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:26.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.562+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac0627b0 con 0x7efeac1430c0 2024-03-25T19:39:26.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.818+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efea40020c0 con 0x7efeac1430c0 2024-03-25T19:39:26.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.818+0000 7efea37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:26.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.818+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7efea4004d00 con 0x7efeac1430c0 2024-03-25T19:39:26.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.818+0000 7efea37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:26.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.818+0000 7efea37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.818+0000 7efea37fe640 10 monclient: dump: 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:26.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.818+0000 7efea9ffb640 4 set_mon_vals no callback set 2024-03-25T19:39:26.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.819+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7efea401ad50 con 0x7efeac1430c0 2024-03-25T19:39:26.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.820+0000 7efeaaffd640 10 monclient: get_auth_request con 0x7efe8c06dcc0 auth_method 0 2024-03-25T19:39:26.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.820+0000 7efea37fe640 10 client.4743.objecter ms_handle_connect 0x7efe8c06dcc0 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7efea4054ac0 con 0x7efeac1430c0 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 10 client.4743.objecter ms_dispatch 0x7efeac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 3 client.4743.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 3 client.4743.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 20 client.4743.objecter dump_active .. 0 homeless 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efea4050400 con 0x7efeac1430c0 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:26.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:26.824+0000 7efea37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:27.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.054+0000 7efeb15e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:27.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.054+0000 7efeb15e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:27.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.054+0000 7efeb15e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:27.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.054+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efeac13e250 con 0x7efeac1430c0 2024-03-25T19:39:27.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.092+0000 7efea37fe640 1 -- 172.21.15.134:0/3376408985 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7efea4052500 con 0x7efeac1430c0 2024-03-25T19:39:27.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.092+0000 7efea37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:27.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.092+0000 7efea37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:27.092 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:39:27.092 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 10 librados: watch_flush enter 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 10 librados: watch_flush exit 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 20 client.4743.objecter shutdown clearing up homeless session... 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 10 client.4743.objecter successfully canceled tick 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 >> v1:172.21.15.159:6800/1875874887 conn(0x7efe8c06dcc0 legacy=0x7efe8c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 10 monclient: shutdown 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.094+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 >> v1:172.21.15.134:6789/0 conn(0x7efeac1430c0 legacy=0x7efeac1db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.095+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 shutdown_connections 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.095+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 >> 172.21.15.134:0/3376408985 conn(0x7efeac0b7ea0 msgr2=0x7efeac14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.095+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 shutdown_connections 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.095+0000 7efeb15e4640 1 -- 172.21.15.134:0/3376408985 wait complete. 2024-03-25T19:39:27.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:27.095+0000 7efeb15e4640 1 librados: shutdown 2024-03-25T19:39:27.106 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:31.642 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:36.314 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.650 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:36.322 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.676 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:36.348 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.704 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:36.351 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.708 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:36.355 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.711 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:36.383 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.717 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:36.390 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.741 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:36.388 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:31.757 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:36.430 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:33.913 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:34.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.029+0000 7fdfbcc8d640 10 monclient: get_monmap_and_config 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.029+0000 7fdfbcc8d640 10 monclient: build_initial_monmap 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.029+0000 7fdfbcc8d640 10 monclient: monmap: 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:39:34.030196+0000 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:39:34.030196+0000 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:39:34.029 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.029+0000 7fdfbcc8d640 1 Processor -- start 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.029+0000 7fdfbcc8d640 1 -- start start 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.029+0000 7fdfbcc8d640 10 monclient: init 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.030+0000 7fdfbcc8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.030+0000 7fdfbcc8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.030+0000 7fdfbcc8d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb80b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.031+0000 7fdfbcc8d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb80b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:34.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.031+0000 7fdfbcc8d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb814be90 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:34.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.031+0000 7fdfbcc8d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80af940 con 0x7fdfb80b4d60 2024-03-25T19:39:34.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.031+0000 7fdfbcc8d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80b0a70 con 0x7fdfb80b1730 2024-03-25T19:39:34.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.031+0000 7fdfbcc8d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80b0d50 con 0x7fdfb814be90 2024-03-25T19:39:34.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.031+0000 7fdfbcc8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:34.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.031+0000 7fdfbcc8d640 10 monclient(hunting): authenticate will time out at 6606.189852s 2024-03-25T19:39:34.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.041+0000 7fdfb6d9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fdfb814be90 0x7fdfb814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:35562/0 (socket says 172.21.15.134:35562) 2024-03-25T19:39:34.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.041+0000 7fdfb6d9d640 1 -- 172.21.15.134:0/3255170948 learned_addr learned my addr 172.21.15.134:0/3255170948 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:34.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.052+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056176460 0 0) 0x7fdfb80b0a70 con 0x7fdfb80b1730 2024-03-25T19:39:34.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.052+0000 7fdfb559a640 10 monclient(hunting): _negotiate 2024-03-25T19:39:34.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.052+0000 7fdfb559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:34.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.052+0000 7fdfb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:34.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.052+0000 7fdfb559a640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:39:34.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.052+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa0003520 con 0x7fdfb80b1730 2024-03-25T19:39:34.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2192928491 0 0) 0x7fdfa0003520 con 0x7fdfb80b1730 2024-03-25T19:39:34.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 >> v1:172.21.15.134:6790/0 conn(0x7fdfb814be90 legacy=0x7fdfb814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:34.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 >> v1:172.21.15.134:6789/0 conn(0x7fdfb80b4d60 legacy=0x7fdfb813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:39:34.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 10 monclient: _finish_hunting 0 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 1 monclient: found mon.noname-c 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb80b1030 con 0x7fdfb80b1730 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 10 monclient: _finish_auth 0 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 10 monclient: _check_auth_tickets 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdfac001690 con 0x7fdfb80b1730 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfb559a640 10 monclient: dump: 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:34.060 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfbcc8d640 5 monclient: authenticate success, global_id 4708 2024-03-25T19:39:34.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.059+0000 7fdfbcc8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfac001860 con 0x7fdfb80b1730 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfb559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfb559a640 1 -- 172.21.15.134:0/3255170948 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdfac001cb0 con 0x7fdfb80b1730 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfbcc8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfbcc8d640 4 set_mon_vals no callback set 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfb559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.067+0000 7fdfb559a640 10 monclient: dump: 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:34.067 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:34.068 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:34.068 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:34.068 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:34.068 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.068+0000 7fdfbcc8d640 10 monclient: shutdown 2024-03-25T19:39:34.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.068+0000 7fdfbcc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:34.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.068+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3255170948 >> v1:172.21.15.159:6789/0 conn(0x7fdfb80b1730 legacy=0x7fdfb80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:34.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.068+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3255170948 shutdown_connections 2024-03-25T19:39:34.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.068+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3255170948 >> 172.21.15.134:0/3255170948 conn(0x7fdfb80617a0 msgr2=0x7fdfb8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.068+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3255170948 shutdown_connections 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.068+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3255170948 wait complete. 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 10 monclient: build_initial_monmap 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 10 monclient: monmap: 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:39:34.070 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 1 librados: starting msgr at 2024-03-25T19:39:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 1 librados: starting objecter 2024-03-25T19:39:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 1 Processor -- start 2024-03-25T19:39:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 1 -- start start 2024-03-25T19:39:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 1 librados: setting wanted keys 2024-03-25T19:39:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 1 librados: calling monclient init 2024-03-25T19:39:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.069+0000 7fdfbcc8d640 10 monclient: init 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb80b1730 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb80b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb814be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80aab40 con 0x7fdfb814be90 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80aae20 con 0x7fdfb80b4d60 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80a9ac0 con 0x7fdfb80b1730 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:34.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.070+0000 7fdfbcc8d640 10 monclient(hunting): authenticate will time out at 6606.229133s 2024-03-25T19:39:34.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.078+0000 7fdfb659c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fdfb80b1730 0x7fdfb8137f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:35602/0 (socket says 172.21.15.134:35602) 2024-03-25T19:39:34.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.078+0000 7fdfb659c640 1 -- 172.21.15.134:0/3547198915 learned_addr learned my addr 172.21.15.134:0/3547198915 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:34.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.079+0000 7fdfa6ffd640 10 client.?.objecter ms_handle_connect 0x7fdfb80b1730 2024-03-25T19:39:34.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.079+0000 7fdfa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:34.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.087+0000 7fdfa6ffd640 10 client.?.objecter ms_handle_connect 0x7fdfb814be90 2024-03-25T19:39:34.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.087+0000 7fdfa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:34.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.087+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204875654 0 0) 0x7fdfb80a9ac0 con 0x7fdfb80b1730 2024-03-25T19:39:34.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.087+0000 7fdfa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:34.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.087+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:34.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.088+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:34.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.088+0000 7fdfa6ffd640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:39:34.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.088+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf98004eb0 con 0x7fdfb80b1730 2024-03-25T19:39:34.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.088+0000 7fdfa6ffd640 10 client.?.objecter ms_handle_connect 0x7fdfb80b4d60 2024-03-25T19:39:34.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.088+0000 7fdfa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:34.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.101+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919883207 0 0) 0x7fdfb80aae20 con 0x7fdfb80b4d60 2024-03-25T19:39:34.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.101+0000 7fdfa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:34.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.101+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:34.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.101+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:34.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.101+0000 7fdfa6ffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:39:34.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.101+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb80a9ac0 con 0x7fdfb80b4d60 2024-03-25T19:39:34.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567593991 0 0) 0x7fdfb80aab40 con 0x7fdfb814be90 2024-03-25T19:39:34.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 10 monclient(hunting): my global_id is 4746 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb80aae20 con 0x7fdfb814be90 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 886115521 0 0) 0x7fdf98004eb0 con 0x7fdfb80b1730 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfb80aab40 con 0x7fdfb80b1730 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdfa8004080 con 0x7fdfb80b1730 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.108+0000 7fdfa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.109+0000 7fdfa6ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.109 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.110 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:34.110 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:34.110 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:34.110 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:34.110 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:34.110 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.109+0000 7fdfa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.122+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1819922169 0 0) 0x7fdfb80aae20 con 0x7fdfb814be90 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.122+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf98004eb0 con 0x7fdfb814be90 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.122+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdfb0001a60 con 0x7fdfb814be90 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.122+0000 7fdfa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.122+0000 7fdfa6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.122+0000 7fdfa6ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:34.122 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.123 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.123 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:34.123 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:34.123 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:34.123 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:34.123 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:34.123 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.126+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3362707394 0 0) 0x7fdfb80a9ac0 con 0x7fdfb80b4d60 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.126+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfb80aae20 con 0x7fdfb80b4d60 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.126+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdfac001b70 con 0x7fdfb80b4d60 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.126+0000 7fdfa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.126+0000 7fdfa6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.126+0000 7fdfa6ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:34.127 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 502777263 0 0) 0x7fdfb80aab40 con 0x7fdfb80b1730 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 >> v1:172.21.15.159:6789/0 conn(0x7fdfb80b4d60 legacy=0x7fdfb81d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 >> v1:172.21.15.134:6789/0 conn(0x7fdfb814be90 legacy=0x7fdfb81d8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 1 monclient: found mon.c 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb80a9c50 con 0x7fdfb80b1730 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:34.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 5 monclient: authenticate success, global_id 4721 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 10 monclient: _renew_subs 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb8143310 con 0x7fdfb80b1730 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 10 client.4721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 10 monclient: _renew_subs 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb8140560 con 0x7fdfb80b1730 2024-03-25T19:39:34.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.127+0000 7fdfbcc8d640 1 librados: init done 2024-03-25T19:39:34.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.128+0000 7fdfbcc8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:34.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.128+0000 7fdfbcc8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:34.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.128+0000 7fdfbcc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:34.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.128+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb80a9710 con 0x7fdfb80b1730 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.460+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfa8000ed0 con 0x7fdfb80b1730 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.460+0000 7fdfa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.460+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fdfa80012a0 con 0x7fdfb80b1730 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.460+0000 7fdfa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.460+0000 7fdfa6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.460+0000 7fdfa6ffd640 10 monclient: dump: 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:34.460 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.460+0000 7fdfa7fff640 4 set_mon_vals no callback set 2024-03-25T19:39:34.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.461+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fdfa80014c0 con 0x7fdfb80b1730 2024-03-25T19:39:34.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.462+0000 7fdfb5d9b640 10 monclient: get_auth_request con 0x7fdf9806e0f0 auth_method 0 2024-03-25T19:39:34.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.462+0000 7fdfa6ffd640 10 client.4721.objecter ms_handle_connect 0x7fdf9806e0f0 2024-03-25T19:39:34.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7fdfa8054780 con 0x7fdfb80b1730 2024-03-25T19:39:34.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 10 client.4721.objecter ms_dispatch 0x7fdfb8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:34.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 3 client.4721.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:34.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 3 client.4721.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:34.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 20 client.4721.objecter dump_active .. 0 homeless 2024-03-25T19:39:34.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfa8050290 con 0x7fdfb80b1730 2024-03-25T19:39:34.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:34.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.465+0000 7fdfa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:34.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.671+0000 7fdfbcc8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:34.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.671+0000 7fdfbcc8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:34.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.671+0000 7fdfbcc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T19:39:34.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.671+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfb80627b0 con 0x7fdfb80b1730 2024-03-25T19:39:34.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.707+0000 7fdfa6ffd640 1 -- 172.21.15.134:0/3547198915 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7fdfa80521c0 con 0x7fdfb80b1730 2024-03-25T19:39:34.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.707+0000 7fdfa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:34.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.707+0000 7fdfa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:34.707 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:39:34.707 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:34.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 10 librados: watch_flush enter 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 10 librados: watch_flush exit 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 20 client.4721.objecter shutdown clearing up homeless session... 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 10 client.4721.objecter successfully canceled tick 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 >> v1:172.21.15.159:6800/1875874887 conn(0x7fdf9806e0f0 legacy=0x7fdf980705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 10 monclient: shutdown 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 >> v1:172.21.15.134:6790/0 conn(0x7fdfb80b1730 legacy=0x7fdfb8137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 shutdown_connections 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 >> 172.21.15.134:0/3547198915 conn(0x7fdfb80617a0 msgr2=0x7fdfb813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 shutdown_connections 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 1 -- 172.21.15.134:0/3547198915 wait complete. 2024-03-25T19:39:34.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:34.709+0000 7fdfbcc8d640 1 librados: shutdown 2024-03-25T19:39:34.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:41.528 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:41.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.648+0000 7f8e8d64e640 10 monclient: get_monmap_and_config 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.648+0000 7f8e8d64e640 10 monclient: build_initial_monmap 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.648+0000 7f8e8d64e640 10 monclient: monmap: 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:39:41.649266+0000 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:39:41.649266+0000 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:39:41.648 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:41.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.649+0000 7f8e8d64e640 1 Processor -- start 2024-03-25T19:39:41.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.649+0000 7f8e8d64e640 1 -- start start 2024-03-25T19:39:41.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.649+0000 7f8e8d64e640 10 monclient: init 2024-03-25T19:39:41.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.649+0000 7f8e8d64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.649+0000 7f8e8d64e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.649+0000 7f8e8d64e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e880705b0 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.649+0000 7f8e8d64e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e881402d0 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.650+0000 7f8e8d64e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e88149ae0 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.650+0000 7f8e8d64e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e880718c0 con 0x7f8e881402d0 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.650+0000 7f8e8d64e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e880714c0 con 0x7f8e88149ae0 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.650+0000 7f8e8d64e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e88072b50 con 0x7f8e880705b0 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.650+0000 7f8e8d64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:41.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.650+0000 7f8e8d64e640 10 monclient(hunting): authenticate will time out at 6613.808673s 2024-03-25T19:39:41.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.661+0000 7f8e867fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8e881402d0 0x7f8e88140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:38634/0 (socket says 172.21.15.134:38634) 2024-03-25T19:39:41.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.661+0000 7f8e867fc640 1 -- 172.21.15.134:0/2522082121 learned_addr learned my addr 172.21.15.134:0/2522082121 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:41.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.677+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803843367 0 0) 0x7f8e88072b50 con 0x7f8e880705b0 2024-03-25T19:39:41.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.677+0000 7f8e85ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:41.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.677+0000 7f8e85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:41.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.677+0000 7f8e85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:41.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.677+0000 7f8e85ffb640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:39:41.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.677+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e70003520 con 0x7f8e880705b0 2024-03-25T19:39:41.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.680+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759319657 0 0) 0x7f8e880714c0 con 0x7f8e88149ae0 2024-03-25T19:39:41.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.680+0000 7f8e85ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:41.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.680+0000 7f8e85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:41.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.680+0000 7f8e85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:41.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.680+0000 7f8e85ffb640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:39:41.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.680+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e88072b50 con 0x7f8e88149ae0 2024-03-25T19:39:41.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.683+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398455727 0 0) 0x7f8e880718c0 con 0x7f8e881402d0 2024-03-25T19:39:41.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.683+0000 7f8e85ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:41.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.683+0000 7f8e85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:41.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.683+0000 7f8e85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:41.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.683+0000 7f8e85ffb640 10 monclient(hunting): my global_id is 4749 2024-03-25T19:39:41.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.683+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e880714c0 con 0x7f8e881402d0 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538688471 0 0) 0x7f8e70003520 con 0x7f8e880705b0 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 >> v1:172.21.15.159:6789/0 conn(0x7f8e88149ae0 legacy=0x7f8e8814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 >> v1:172.21.15.134:6789/0 conn(0x7f8e881402d0 legacy=0x7f8e88140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 1 monclient: found mon.noname-b 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8813dd10 con 0x7f8e880705b0 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:41.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8e78001690 con 0x7f8e880705b0 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e85ffb640 10 monclient: dump: 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:41.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e8d64e640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:39:41.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.691+0000 7f8e8d64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:41.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e780017f0 con 0x7f8e880705b0 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e85ffb640 1 -- 172.21.15.134:0/2522082121 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8e78001c00 con 0x7f8e880705b0 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e85ffb640 10 monclient: dump: 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:41.705 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:41.706 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:41.706 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:41.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e8d64e640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:41.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.705+0000 7f8e8d64e640 4 set_mon_vals no callback set 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.706+0000 7f8e8d64e640 10 monclient: shutdown 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.706+0000 7f8e8d64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.706+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2522082121 >> v1:172.21.15.134:6790/0 conn(0x7f8e880705b0 legacy=0x7f8e8813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.706+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2522082121 shutdown_connections 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.706+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2522082121 >> 172.21.15.134:0/2522082121 conn(0x7f8e880b5d10 msgr2=0x7f8e880b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.706+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2522082121 shutdown_connections 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.706+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2522082121 wait complete. 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.707+0000 7f8e8d64e640 10 monclient: build_initial_monmap 2024-03-25T19:39:41.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.707+0000 7f8e8d64e640 10 monclient: monmap: 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.707+0000 7f8e8d64e640 1 librados: starting msgr at 2024-03-25T19:39:41.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.707+0000 7f8e8d64e640 1 librados: starting objecter 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 1 Processor -- start 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 1 -- start start 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 1 librados: setting wanted keys 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 1 librados: calling monclient init 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 10 monclient: init 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e880705b0 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.708+0000 7f8e8d64e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e881402d0 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.709+0000 7f8e8d64e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e88149ae0 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.709+0000 7f8e8d64e640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8806bd90 con 0x7f8e880705b0 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.709+0000 7f8e8d64e640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8806d4e0 con 0x7f8e88149ae0 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.709+0000 7f8e8d64e640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8806d970 con 0x7f8e881402d0 2024-03-25T19:39:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.709+0000 7f8e8d64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:41.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.709+0000 7f8e8d64e640 10 monclient(hunting): authenticate will time out at 6613.867786s 2024-03-25T19:39:41.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.723+0000 7f8e877fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f8e88149ae0 0x7f8e881e2280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:40852/0 (socket says 172.21.15.134:40852) 2024-03-25T19:39:41.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.723+0000 7f8e877fe640 1 -- 172.21.15.134:0/2280763504 learned_addr learned my addr 172.21.15.134:0/2280763504 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:41.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.724+0000 7f8e6f7fe640 10 client.?.objecter ms_handle_connect 0x7f8e88149ae0 2024-03-25T19:39:41.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.724+0000 7f8e6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:41.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.730+0000 7f8e6f7fe640 10 client.?.objecter ms_handle_connect 0x7f8e881402d0 2024-03-25T19:39:41.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.730+0000 7f8e6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:41.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.732+0000 7f8e6f7fe640 10 client.?.objecter ms_handle_connect 0x7f8e880705b0 2024-03-25T19:39:41.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.732+0000 7f8e6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:41.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.736+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969763940 0 0) 0x7f8e8806d4e0 con 0x7f8e88149ae0 2024-03-25T19:39:41.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.736+0000 7f8e6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:41.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.736+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:41.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.736+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:41.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.736+0000 7f8e6f7fe640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:39:41.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.736+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e74004eb0 con 0x7f8e88149ae0 2024-03-25T19:39:41.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.746+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821962416 0 0) 0x7f8e8806d970 con 0x7f8e881402d0 2024-03-25T19:39:41.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.746+0000 7f8e6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:41.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.746+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:41.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.746+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:41.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.746+0000 7f8e6f7fe640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:39:41.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.746+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8806d4e0 con 0x7f8e881402d0 2024-03-25T19:39:41.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.747+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961962526 0 0) 0x7f8e8806bd90 con 0x7f8e880705b0 2024-03-25T19:39:41.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.747+0000 7f8e6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:41.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.747+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:41.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.747+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:41.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.747+0000 7f8e6f7fe640 10 monclient(hunting): my global_id is 4752 2024-03-25T19:39:41.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.747+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8806d970 con 0x7f8e880705b0 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.771+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3948035917 0 0) 0x7f8e74004eb0 con 0x7f8e88149ae0 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.771+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e8806bd90 con 0x7f8e88149ae0 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.771+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8e80001c30 con 0x7f8e88149ae0 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.771+0000 7f8e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.771+0000 7f8e6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.771+0000 7f8e6f7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:41.771 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:41.772 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:41.772 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:41.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.771+0000 7f8e6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.781+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2122427317 0 0) 0x7f8e8806d4e0 con 0x7f8e881402d0 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.781+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e74004eb0 con 0x7f8e881402d0 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.781+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8e7c0018b0 con 0x7f8e881402d0 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.781+0000 7f8e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.781+0000 7f8e6f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.781+0000 7f8e6f7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:41.781 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:41.782 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:41.782 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:41.782 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:41.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.782+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071067693 0 0) 0x7f8e8806d970 con 0x7f8e880705b0 2024-03-25T19:39:41.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.782+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e8806d4e0 con 0x7f8e880705b0 2024-03-25T19:39:41.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.782+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8e78001b00 con 0x7f8e880705b0 2024-03-25T19:39:41.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.782+0000 7f8e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.782+0000 7f8e6f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.782+0000 7f8e6f7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:41.783 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 403072173 0 0) 0x7f8e8806bd90 con 0x7f8e88149ae0 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 >> v1:172.21.15.134:6790/0 conn(0x7f8e881402d0 legacy=0x7f8e881dace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 >> v1:172.21.15.134:6789/0 conn(0x7f8e880705b0 legacy=0x7f8e8806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 1 monclient: found mon.b 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:41.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8806d020 con 0x7f8e88149ae0 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 5 monclient: authenticate success, global_id 4717 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 10 monclient: _renew_subs 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e88148ee0 con 0x7f8e88149ae0 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 10 monclient: _renew_subs 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e881db3f0 con 0x7f8e88149ae0 2024-03-25T19:39:41.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.790+0000 7f8e8d64e640 1 librados: init done 2024-03-25T19:39:41.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.791+0000 7f8e8d64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:41.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.791+0000 7f8e8d64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:41.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.791+0000 7f8e8d64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:41.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:41.791+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e880627b0 con 0x7f8e88149ae0 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.124+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e80003990 con 0x7f8e88149ae0 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.125+0000 7f8e6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.125+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7f8e80002100 con 0x7f8e88149ae0 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.125+0000 7f8e6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.125+0000 7f8e6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.125+0000 7f8e6f7fe640 10 monclient: dump: 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:42.125 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:42.126 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:42.126 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:42.126 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:42.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.125+0000 7f8e857fa640 4 set_mon_vals no callback set 2024-03-25T19:39:42.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.126+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7f8e80003e20 con 0x7f8e88149ae0 2024-03-25T19:39:42.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.126+0000 7f8e86ffd640 10 monclient: get_auth_request con 0x7f8e7406e0f0 auth_method 0 2024-03-25T19:39:42.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.126+0000 7f8e6f7fe640 10 client.4717.objecter ms_handle_connect 0x7f8e7406e0f0 2024-03-25T19:39:42.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.130+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7f8e80054a20 con 0x7f8e88149ae0 2024-03-25T19:39:42.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.130+0000 7f8e6f7fe640 10 client.4717.objecter ms_dispatch 0x7f8e88000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:42.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.130+0000 7f8e6f7fe640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:42.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.130+0000 7f8e6f7fe640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:42.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.130+0000 7f8e6f7fe640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T19:39:42.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.131+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e80050530 con 0x7f8e88149ae0 2024-03-25T19:39:42.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.131+0000 7f8e6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:42.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.131+0000 7f8e6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:42.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.384+0000 7f8e8d64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:42.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.384+0000 7f8e8d64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:42.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.384+0000 7f8e8d64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T19:39:42.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.384+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e880b5880 con 0x7f8e88149ae0 2024-03-25T19:39:42.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.423+0000 7f8e6f7fe640 1 -- 172.21.15.134:0/2280763504 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7f8e80052460 con 0x7f8e88149ae0 2024-03-25T19:39:42.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.423+0000 7f8e6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:42.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.423+0000 7f8e6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:42.423 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:39:42.423 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.425+0000 7f8e8d64e640 10 librados: watch_flush enter 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.425+0000 7f8e8d64e640 10 librados: watch_flush exit 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.425+0000 7f8e8d64e640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.425+0000 7f8e8d64e640 10 client.4717.objecter successfully canceled tick 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 >> v1:172.21.15.159:6800/1875874887 conn(0x7f8e7406e0f0 legacy=0x7f8e740705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 10 monclient: shutdown 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:42.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 >> v1:172.21.15.159:6789/0 conn(0x7f8e88149ae0 legacy=0x7f8e881e2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:42.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 shutdown_connections 2024-03-25T19:39:42.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 >> 172.21.15.134:0/2280763504 conn(0x7f8e880b5d10 msgr2=0x7f8e880b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:42.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 shutdown_connections 2024-03-25T19:39:42.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 1 -- 172.21.15.134:0/2280763504 wait complete. 2024-03-25T19:39:42.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:42.426+0000 7f8e8d64e640 1 librados: shutdown 2024-03-25T19:39:42.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:49.245 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.364+0000 7fb7b46ba640 10 monclient: get_monmap_and_config 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.364+0000 7fb7b46ba640 10 monclient: build_initial_monmap 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.364+0000 7fb7b46ba640 10 monclient: monmap: 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:39:49.365971+0000 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:39:49.365971+0000 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:49.365 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:49.366 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:49.366 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T19:39:49.366 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T19:39:49.366 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.365+0000 7fb7b46ba640 1 Processor -- start 2024-03-25T19:39:49.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.365+0000 7fb7b46ba640 1 -- start start 2024-03-25T19:39:49.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.365+0000 7fb7b46ba640 10 monclient: init 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 10 monclient(hunting): picked mon.noname-b con 0x7fb7ac0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 10 monclient(hunting): picked mon.noname-c con 0x7fb7ac0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 10 monclient(hunting): picked mon.noname-a con 0x7fb7ac14be90 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0af940 con 0x7fb7ac14be90 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0b0a70 con 0x7fb7ac0b1730 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0b0d50 con 0x7fb7ac0b4d60 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:49.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.366+0000 7fb7b46ba640 10 monclient(hunting): authenticate will time out at 6621.525587s 2024-03-25T19:39:49.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.371+0000 7fb7b1c55640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb7ac0b1730 0x7fb7ac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:45774/0 (socket says 172.21.15.134:45774) 2024-03-25T19:39:49.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.371+0000 7fb7b1c55640 1 -- 172.21.15.134:0/4198802213 learned_addr learned my addr 172.21.15.134:0/4198802213 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:49.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.382+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851214354 0 0) 0x7fb7ac0b0a70 con 0x7fb7ac0b1730 2024-03-25T19:39:49.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.382+0000 7fb7b1454640 10 monclient(hunting): _negotiate 2024-03-25T19:39:49.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.382+0000 7fb7b1454640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:49.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.382+0000 7fb7b1454640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:49.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.382+0000 7fb7b1454640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:39:49.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.382+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb798003520 con 0x7fb7ac0b1730 2024-03-25T19:39:49.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.387+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735242277 0 0) 0x7fb7ac0af940 con 0x7fb7ac14be90 2024-03-25T19:39:49.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.387+0000 7fb7b1454640 10 monclient(hunting): _negotiate 2024-03-25T19:39:49.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.387+0000 7fb7b1454640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:49.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.387+0000 7fb7b1454640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:49.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.387+0000 7fb7b1454640 10 monclient(hunting): my global_id is 4755 2024-03-25T19:39:49.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.387+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac0b0a70 con 0x7fb7ac14be90 2024-03-25T19:39:49.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589683650 0 0) 0x7fb798003520 con 0x7fb7ac0b1730 2024-03-25T19:39:49.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 >> v1:172.21.15.134:6790/0 conn(0x7fb7ac0b4d60 legacy=0x7fb7ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:49.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 >> v1:172.21.15.134:6789/0 conn(0x7fb7ac14be90 legacy=0x7fb7ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 10 monclient: _finish_hunting 0 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 1 monclient: found mon.noname-c 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac0b1030 con 0x7fb7ac0b1730 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 10 monclient: _finish_auth 0 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 10 monclient: _check_auth_tickets 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb79c001690 con 0x7fb7ac0b1730 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b1454640 10 monclient: dump: 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:49.395 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b46ba640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:39:49.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.394+0000 7fb7b46ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:49.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb79c0017f0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b1454640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:49.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b1454640 1 -- 172.21.15.134:0/4198802213 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb79c001c00 con 0x7fb7ac0b1730 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b46ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b46ba640 4 set_mon_vals no callback set 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b1454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b1454640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.402+0000 7fb7b1454640 10 monclient: dump: 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:49.402 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.403+0000 7fb7b46ba640 10 monclient: shutdown 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.403+0000 7fb7b46ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.403+0000 7fb7b46ba640 1 -- 172.21.15.134:0/4198802213 >> v1:172.21.15.159:6789/0 conn(0x7fb7ac0b1730 legacy=0x7fb7ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.403+0000 7fb7b46ba640 1 -- 172.21.15.134:0/4198802213 shutdown_connections 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.403+0000 7fb7b46ba640 1 -- 172.21.15.134:0/4198802213 >> 172.21.15.134:0/4198802213 conn(0x7fb7ac0617a0 msgr2=0x7fb7ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.403+0000 7fb7b46ba640 1 -- 172.21.15.134:0/4198802213 shutdown_connections 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.403+0000 7fb7b46ba640 1 -- 172.21.15.134:0/4198802213 wait complete. 2024-03-25T19:39:49.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 10 monclient: build_initial_monmap 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 10 monclient: monmap: 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 1 librados: starting msgr at 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 1 librados: starting objecter 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 1 Processor -- start 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 1 -- start start 2024-03-25T19:39:49.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 1 librados: setting wanted keys 2024-03-25T19:39:49.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 1 librados: calling monclient init 2024-03-25T19:39:49.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.404+0000 7fb7b46ba640 10 monclient: init 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 10 monclient(hunting): picked mon.noname-a con 0x7fb7ac0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 10 monclient(hunting): picked mon.noname-c con 0x7fb7ac0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 10 monclient(hunting): picked mon.noname-b con 0x7fb7ac14be90 addr v1:172.21.15.159:6789/0 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac13ada0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac1d8360 con 0x7fb7ac14be90 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac1d84f0 con 0x7fb7ac0b4d60 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:49.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.405+0000 7fb7b46ba640 10 monclient(hunting): authenticate will time out at 6621.564497s 2024-03-25T19:39:49.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.416+0000 7fb7b2456640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb7ac0b1730 0x7fb7ac1d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:43496/0 (socket says 172.21.15.134:43496) 2024-03-25T19:39:49.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.416+0000 7fb7b2456640 1 -- 172.21.15.134:0/3549798414 learned_addr learned my addr 172.21.15.134:0/3549798414 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T19:39:49.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.419+0000 7fb7a2ffd640 10 client.?.objecter ms_handle_connect 0x7fb7ac0b1730 2024-03-25T19:39:49.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.419+0000 7fb7a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:49.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.429+0000 7fb7a2ffd640 10 client.?.objecter ms_handle_connect 0x7fb7ac0b4d60 2024-03-25T19:39:49.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.429+0000 7fb7a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:49.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.430+0000 7fb7a2ffd640 10 client.?.objecter ms_handle_connect 0x7fb7ac14be90 2024-03-25T19:39:49.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.430+0000 7fb7a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:49.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.431+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985328529 0 0) 0x7fb7ac13ada0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.431+0000 7fb7a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:49.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.431+0000 7fb7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:49.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.431+0000 7fb7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:49.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.431+0000 7fb7a2ffd640 10 monclient(hunting): my global_id is 4758 2024-03-25T19:39:49.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.431+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb790004eb0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242301216 0 0) 0x7fb7ac1d84f0 con 0x7fb7ac0b4d60 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac13ada0 con 0x7fb7ac0b4d60 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202031793 0 0) 0x7fb7ac1d8360 con 0x7fb7ac14be90 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:39:49.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.444+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac1d84f0 con 0x7fb7ac14be90 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.457+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809420600 0 0) 0x7fb790004eb0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.457+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7ac1d8360 con 0x7fb7ac0b1730 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.457+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb7a80019d0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.457+0000 7fb7a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.457+0000 7fb7a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.457+0000 7fb7a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.457 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:49.458 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:49.458 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:49.458 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:49.458 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:49.458 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.457+0000 7fb7a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.472+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888616169 0 0) 0x7fb7ac1d84f0 con 0x7fb7ac14be90 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.472+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb790004eb0 con 0x7fb7ac14be90 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.472+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb7a4001b50 con 0x7fb7ac14be90 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.472+0000 7fb7a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.472+0000 7fb7a2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.472+0000 7fb7a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:49.472 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:49.473 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.477+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3750541621 0 0) 0x7fb7ac13ada0 con 0x7fb7ac0b4d60 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.477+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7ac1d84f0 con 0x7fb7ac0b4d60 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.477+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb79c001b00 con 0x7fb7ac0b4d60 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.477+0000 7fb7a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.477+0000 7fb7a2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.477+0000 7fb7a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:49.479 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1153534740 0 0) 0x7fb7ac1d8360 con 0x7fb7ac0b1730 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 >> v1:172.21.15.134:6790/0 conn(0x7fb7ac0b4d60 legacy=0x7fb7ac134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 >> v1:172.21.15.159:6789/0 conn(0x7fb7ac14be90 legacy=0x7fb7ac1e2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 1 monclient: found mon.a 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac139b10 con 0x7fb7ac0b1730 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:49.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 5 monclient: authenticate success, global_id 4758 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 10 monclient: _renew_subs 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac1355c0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 10 client.4758.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 10 monclient: _renew_subs 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac13b3a0 con 0x7fb7ac0b1730 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.478+0000 7fb7b46ba640 1 librados: init done 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.479+0000 7fb7b46ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.479+0000 7fb7b46ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.479+0000 7fb7b46ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:49.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.479+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac139d80 con 0x7fb7ac0b1730 2024-03-25T19:39:49.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.788+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7a8003610 con 0x7fb7ac0b1730 2024-03-25T19:39:49.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.788+0000 7fb7a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:49.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.788+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 178300990 0 0) 0x7fb7a8003a20 con 0x7fb7ac0b1730 2024-03-25T19:39:49.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.788+0000 7fb7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:49.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.788+0000 7fb7a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:49.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.788+0000 7fb7a2ffd640 10 monclient: dump: 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:fsid 66efaa2b-ef2e-4206-b9a2-97b6c1780495 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T19:27:22.363659+0000 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T19:39:49.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.788+0000 7fb7b0c53640 4 set_mon_vals no callback set 2024-03-25T19:39:49.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.789+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3217384194 0 0) 0x7fb7a8003c40 con 0x7fb7ac0b1730 2024-03-25T19:39:49.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.790+0000 7fb7b1c55640 10 monclient: get_auth_request con 0x7fb79006e0d0 auth_method 0 2024-03-25T19:39:49.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.790+0000 7fb7a2ffd640 10 client.4758.objecter ms_handle_connect 0x7fb79006e0d0 2024-03-25T19:39:49.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3351280626 0 0) 0x7fb7a8054770 con 0x7fb7ac0b1730 2024-03-25T19:39:49.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 10 client.4758.objecter ms_dispatch 0x7fb7ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:49.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 3 client.4758.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:49.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 3 client.4758.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:49.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 20 client.4758.objecter dump_active .. 0 homeless 2024-03-25T19:39:49.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7a8050280 con 0x7fb7ac0b1730 2024-03-25T19:39:49.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:49.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:49.796+0000 7fb7a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:50.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.047+0000 7fb7b46ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:50.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.047+0000 7fb7b46ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:50.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.047+0000 7fb7b46ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T19:39:50.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.047+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7ac0627b0 con 0x7fb7ac0b1730 2024-03-25T19:39:50.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.077+0000 7fb7a2ffd640 1 -- 172.21.15.134:0/3549798414 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4172970710) 0x7fb7a80521b0 con 0x7fb7ac0b1730 2024-03-25T19:39:50.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.077+0000 7fb7a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:50.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.077+0000 7fb7a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:50.077 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T19:39:50.077 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"66efaa2b-ef2e-4206-b9a2-97b6c1780495","created":"2024-03-25T19:28:22.432241+0000","modified":"2024-03-25T19:38:26.333493+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:26.333493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6064f78c-3ecd-4a52-bbbe-608a8fdf3bca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9c95d080-d8a1-4d86-afea-ed5e71b09634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5ac7ba50-abaa-41fc-9f8c-5c18ef03aba9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dfd685c1-d3ec-4f5d-9e86-56b1c412f421","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e799e3d-df93-4507-a38b-5e5b552aefb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"538f3a54-921e-4264-8bfd-11837dc6ce63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"93761c0a-2dcf-45f2-acae-3d602d251675","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"519ac8d9-2636-4167-97a5-3feca8638ff8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:50.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.079+0000 7fb7b46ba640 10 librados: watch_flush enter 2024-03-25T19:39:50.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.079+0000 7fb7b46ba640 10 librados: watch_flush exit 2024-03-25T19:39:50.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.079+0000 7fb7b46ba640 20 client.4758.objecter shutdown clearing up homeless session... 2024-03-25T19:39:50.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.079+0000 7fb7b46ba640 10 client.4758.objecter successfully canceled tick 2024-03-25T19:39:50.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 >> v1:172.21.15.159:6800/1875874887 conn(0x7fb79006e0d0 legacy=0x7fb790070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 10 monclient: shutdown 2024-03-25T19:39:50.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:50.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 >> v1:172.21.15.134:6789/0 conn(0x7fb7ac0b1730 legacy=0x7fb7ac1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 shutdown_connections 2024-03-25T19:39:50.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 >> 172.21.15.134:0/3549798414 conn(0x7fb7ac0617a0 msgr2=0x7fb7ac137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:50.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 shutdown_connections 2024-03-25T19:39:50.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 1 -- 172.21.15.134:0/3549798414 wait complete. 2024-03-25T19:39:50.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T19:39:50.080+0000 7fb7b46ba640 1 librados: shutdown 2024-03-25T19:39:50.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:50.092 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:39:50.094 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:39:50.094 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:39:50.094 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:51.673 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:56.346 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.677 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:56.349 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.761 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:56.434 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.790 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:56.437 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.795 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:56.467 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.796 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:56.443 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.800 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:56.473 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.820 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:56.467 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.840 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:56.512 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:51.940 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:11:23.990 35442== 2024-03-25T19:39:51.940 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:11:23.990 35442== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:51.976 INFO:tasks.ceph.osd.0.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:56.196 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:39:56.197 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:39:56.197 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:39:56.197 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:58.054 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:11:30.096 35444== 2024-03-25T19:39:58.054 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:11:30.096 35444== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:58.089 INFO:tasks.ceph.osd.1.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:02.299 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:40:02.299 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:40:02.299 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:40:02.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:04.171 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:11:36.220 35443== 2024-03-25T19:40:04.171 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:11:36.220 35443== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:04.215 INFO:tasks.ceph.osd.2.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:08.402 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:40:08.402 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:40:08.402 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:40:08.403 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:10.222 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:11:42.278 35441== 2024-03-25T19:40:10.222 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:11:42.278 35441== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:10.266 INFO:tasks.ceph.osd.3.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:11.741 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:16.414 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.745 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:16.417 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.768 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:16.441 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.795 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:16.442 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.797 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:16.470 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.798 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:16.445 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.800 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:16.473 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.823 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:16.496 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:11.829 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:16.476 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:14.504 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:40:14.504 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:40:14.505 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:40:14.505 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:16.582 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:11:48.645 35432== 2024-03-25T19:40:16.582 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:11:48.645 35432== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:16.626 INFO:tasks.ceph.osd.4.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:20.607 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:40:20.608 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:40:20.608 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:40:20.608 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:22.463 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:11:54.524 35433== 2024-03-25T19:40:22.463 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:11:54.524 35433== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:22.507 INFO:tasks.ceph.osd.5.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:26.709 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:40:26.710 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:40:26.710 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:40:26.710 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:28.409 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:12:00.467 35434== 2024-03-25T19:40:28.409 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:12:00.467 35434== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:28.442 INFO:tasks.ceph.osd.6.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:31.738 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:36.411 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.739 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:36.412 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.801 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:36.473 35130== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.835 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:36.481 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.835 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:36.507 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.839 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:36.486 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.840 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:36.512 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.856 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:36.529 35131== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:31.856 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:36.503 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:32.813 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:40:32.813 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:40:32.813 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:40:32.813 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:34.373 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:12:06.431 35435== 2024-03-25T19:40:34.374 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:12:06.431 35435== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:34.403 INFO:tasks.ceph.osd.7.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:38.915 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:40:38.916 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:40:38.916 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:40:38.916 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:40:38.916 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:38.943 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:40:38.943 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:40:38.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:38.971 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:40:38.971 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:40:38.971 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:40:38.972 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:39.001 INFO:tasks.ceph.mon.a.smithi134.stderr:2024-03-25T19:40:38.988+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35096) UID: 0 2024-03-25T19:40:39.001 INFO:tasks.ceph.mon.a.smithi134.stderr:2024-03-25T19:40:38.990+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:40:39.529 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:44.201 35130== 2024-03-25T19:40:39.529 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:44.201 35130== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:39.538 INFO:tasks.ceph.mon.a.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:45.073 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:40:45.074 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:40:45.074 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:40:45.074 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:45.091 INFO:tasks.ceph.mon.c.smithi134.stderr:2024-03-25T19:40:45.079+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35098) UID: 0 2024-03-25T19:40:45.092 INFO:tasks.ceph.mon.c.smithi134.stderr:2024-03-25T19:40:45.080+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:40:45.534 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:50.206 35131== 2024-03-25T19:40:45.534 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:50.206 35131== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:45.544 INFO:tasks.ceph.mon.c.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:51.176 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:40:51.177 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:40:51.177 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:40:51.177 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:51.192 INFO:tasks.ceph.mon.b.smithi159.stderr:2024-03-25T19:40:51.181+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34883) UID: 0 2024-03-25T19:40:51.193 INFO:tasks.ceph.mon.b.smithi159.stderr:2024-03-25T19:40:51.183+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:40:51.702 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:56.348 34887== 2024-03-25T19:40:51.702 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:56.348 34887== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:51.711 INFO:tasks.ceph.mon.b.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:57.280 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:40:57.280 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:40:57.280 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:40:57.281 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:40:57.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:40:57.394 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:57.484 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:40:57.484 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:57.557 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:40:57.557 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:57.648 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:40:57.648 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:57.734 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:40:57.734 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:57.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:40:57.885 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:57.957 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:40:57.957 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:58.015 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:40:58.015 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:58.073 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:40:58.075 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/data/mon.a.tgz 2024-03-25T19:40:58.075 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-25T19:40:58.090 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.OF8VBOPhV5 2024-03-25T19:40:58.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.OF8VBOPhV5 2024-03-25T19:40:58.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.OF8VBOPhV5 2024-03-25T19:40:58.292 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.OF8VBOPhV5 is 134KB 2024-03-25T19:40:58.344 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.OF8VBOPhV5 2024-03-25T19:40:58.358 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/data/mon.c.tgz 2024-03-25T19:40:58.358 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-25T19:40:58.412 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.Uk712lmxvr 2024-03-25T19:40:58.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Uk712lmxvr 2024-03-25T19:40:58.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.Uk712lmxvr 2024-03-25T19:40:58.584 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.Uk712lmxvr is 134KB 2024-03-25T19:40:58.634 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.Uk712lmxvr 2024-03-25T19:40:58.648 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/data/mon.b.tgz 2024-03-25T19:40:58.648 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-25T19:40:58.663 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.zPJ9hx4lby 2024-03-25T19:40:58.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zPJ9hx4lby 2024-03-25T19:40:58.762 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.zPJ9hx4lby 2024-03-25T19:40:58.846 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.zPJ9hx4lby is 134KB 2024-03-25T19:40:58.897 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.zPJ9hx4lby 2024-03-25T19:40:58.911 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:40:58.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:40:58.913 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:40:59.268 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:40:59.477 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:40:59.479 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/remote/smithi134/crash 2024-03-25T19:40:59.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:40:59.507 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/remote/smithi159/crash 2024-03-25T19:40:59.508 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:40:59.534 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:40:59.534 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:40:59.551 DEBUG:teuthology.orchestra.run.smithi159:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:40:59.623 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:40:59.623 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.4.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35082.log 2024-03-25T19:40:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35122.log 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35202.log 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35122.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35082.log.gz 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35202.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35122.log.gz -5 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr: --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35333.log 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.5.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35202.log.gz 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35042.log 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35242.log 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35162.log 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35333.log.gz 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:40:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35242.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35042.log.gz 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35162.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35293.log 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35162.log.gz 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35293.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35293.log.gz 2024-03-25T19:40:59.626 INFO:teuthology.orchestra.run.smithi159.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-03-25T19:40:59.627 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40135.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38740.log 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39156.log 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38740.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41891.log 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38740.log.gz 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39627.log 2024-03-25T19:40:59.628 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39156.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41891.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39627.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41891.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-03-25T19:40:59.629 INFO:teuthology.orchestra.run.smithi134.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40369.log 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41149.log: 74.3% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35589.log 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41500.log 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40369.log.gz 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39784.log 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35589.log.gz 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41500.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39117.log 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41500.log.gz 2024-03-25T19:40:59.630 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37446.log 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39784.log.gz 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40018.log 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.37446.log: gzip -5 87.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39117.log.gz 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: /var/log/ceph/ceph-client.admin.40525.log 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37446.log.gz 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41774.log 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40018.log.gz 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42203.log 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39273.log 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2024-03-25T19:40:59.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42203.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41774.log.gz 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41188.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29272.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39273.log: /var/log/ceph/ceph-client.admin.41188.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42203.log.gz 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39549.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39273.log.gz 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.tmp-client.admin.29272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41227.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41188.log.gz 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29272.log.gz 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/ceph-client.admin.39549.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39353.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39549.log.gz 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38958.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41227.log.gz 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-03-25T19:40:59.632 INFO:teuthology.orchestra.run.smithi134.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39353.log.gz 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38958.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38958.log.gz 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39745.log 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40447.log 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39745.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41735.log 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40447.log: gzip 86.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39745.log.gz 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr: -- /var/log/ceph/ceph-client.admin.36471.log 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42008.log 2024-03-25T19:40:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.36471.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40447.log.gz 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41735.log.gz 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42008.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36471.log.gz 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40603.log 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40057.log 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39038.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42008.log.gz 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39588.log 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38997.log 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40057.log.gz 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39588.log: /var/log/ceph/ceph-client.admin.40642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41578.log 2024-03-25T19:40:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39588.log.gz 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38997.log.gz 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40252.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41578.log.gz 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39510.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35138.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39901.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41422.log 2024-03-25T19:40:59.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35138.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39510.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41695.log 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39901.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.35138.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/ceph-client.admin.41695.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41422.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41695.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41266.log 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40096.log: -- /var/log/ceph/ceph-client.admin.40681.log 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40681.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41266.log.gz 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40759.log: gzip -5 --verbose -- 86.6% /var/log/ceph/ceph-client.admin.39195.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40681.log.gz 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35440.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.3.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35467.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41383.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39195.log.gz 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35467.log: gzip -5 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35440.log.gz 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41383.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35467.log.gz 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41344.log 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39078.log 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40408.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41383.log.gz 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41344.log.gz 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/ceph-client.admin.40330.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39078.log.gz 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39234.log 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39234.log.gz 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr: /var/log/ceph/ceph-client.admin.42047.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40213.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr: -- /var/log/ceph/ceph-client.admin.39666.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42047.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42164.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5/var/log/ceph/ceph-osd.0.log: --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42164.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39666.log.gz 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39979.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42164.log.gz 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41305.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41305.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39979.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41305.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40564.log 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42242.log 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.40564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42242.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40564.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42242.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39823.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2024-03-25T19:40:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41930.log 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42086.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39940.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41930.log.gz 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39862.log 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39432.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42125.log 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39312.log 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42125.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39862.log.gz 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39312.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42125.log.gz 2024-03-25T19:40:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mon.c.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39312.log.gz 2024-03-25T19:40:59.690 INFO:teuthology.orchestra.run.smithi159.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:41:00.399 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:41:00.409 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:41:00.463 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:41:00.508 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:41:00.529 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:41:00.727 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:41:00.732 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:41:00.812 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:41:04.759 INFO:teuthology.orchestra.run.smithi134.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:41:04.832 INFO:teuthology.orchestra.run.smithi159.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:41:04.834 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T19:41:04.834 INFO:teuthology.orchestra.run.smithi159.stderr:real 0m5.240s 2024-03-25T19:41:04.834 INFO:teuthology.orchestra.run.smithi159.stderr:user 0m8.905s 2024-03-25T19:41:04.834 INFO:teuthology.orchestra.run.smithi159.stderr:sys 0m0.297s 2024-03-25T19:41:04.975 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:41:04.977 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T19:41:04.977 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m5.414s 2024-03-25T19:41:04.977 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m13.979s 2024-03-25T19:41:04.977 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.565s 2024-03-25T19:41:04.977 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:41:04.977 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/remote/smithi134/log 2024-03-25T19:41:04.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:41:07.404 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/remote/smithi159/log 2024-03-25T19:41:07.405 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:41:08.919 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:41:09.128 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ac3ddef8859e4e748788aada5fc51344 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:41:09.133 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:41:09.144 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:41:09.158 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:41:09.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:41:09.173 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:41:09.257 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:41:09.269 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:41:09.270 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:41:09.273 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:41:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:41:09.289 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:41:09.291 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:41:09.291 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-25T19:41:09.291 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 -257us[ -257us] +/- 68ms 2024-03-25T19:41:09.291 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -289us[ -294us] +/- 54ms 2024-03-25T19:41:09.291 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +804us[ +804us] +/- 90ms 2024-03-25T19:41:09.292 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:41:09.294 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:41:09.294 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-25T19:41:09.294 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 153 -292us[ -321us] +/- 66ms 2024-03-25T19:41:09.294 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv02.front.sepia.ceph.com 3 7 377 35 -371us[ -400us] +/- 54ms 2024-03-25T19:41:09.294 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 32 +707us[ +707us] +/- 90ms 2024-03-25T19:41:09.294 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:41:09.295 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:41:09.306 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:41:09.308 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:41:09.322 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:41:09.402 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:41:09.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:41:09.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:09.503 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:41:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:09.535 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:41:09.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:41:09.559 INFO:teuthology.task.internal:Duration was 6319.620281 seconds 2024-03-25T19:41:09.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:41:09.609 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:41:09.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:41:09.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:41:09.654 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:41:09.655 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:41:09.933 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:41:09.934 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T19:41:09.934 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:41:09.955 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T19:41:09.955 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:41:09.998 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:41:09.999 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:41:10.001 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:41:10.068 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:41:10.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:41:10.072 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:41:10.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:41:10.224 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:41:10.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:41:10.251 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:41:10.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:41:10.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:10.294 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:10.317 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-03-25T19:41:10.344 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-03-25T19:41:10.366 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:10.404 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:10.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:41:10.434 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:41:10.434 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/remote/smithi134 2024-03-25T19:41:10.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:41:10.481 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314/remote/smithi159 2024-03-25T19:41:10.481 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:41:10.515 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:41:10.515 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:41:10.518 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:41:10.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:41:10.576 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:41:10.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:41:10.659 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:41:10.660 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:41:10.662 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:41:10.674 INFO:teuthology.orchestra.run.smithi134.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:41 /home/ubuntu/cephtest 2024-03-25T19:41:10.677 INFO:teuthology.orchestra.run.smithi159.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:41 /home/ubuntu/cephtest 2024-03-25T19:41:10.678 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:41:10.691 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:41:10.719 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:41:10.746 DEBUG:teuthology.nuke:shortname: smithi134 2024-03-25T19:41:10.746 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:41:10.772 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.417185', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA2kYWyeLWstO2L6gPtqoaupxwqZeJGQBC0Y1ga9le3tLy/Ar4QYnSk9MoRJYwdqZ0bE0cpZ8jOK54Ch+/4cOEA='} 2024-03-25T19:41:10.775 DEBUG:teuthology.nuke:shortname: smithi159 2024-03-25T19:41:10.775 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:41:10.799 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.415660', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGX+Tafmgyf1WOPecYLO/4QNjiuEFjSTSXj4gKaNTmmczFBACEc92+GuTsy4TP3mDPyVmz5aT9XP/oSlYbNjtE='} 2024-03-25T19:41:10.826 INFO:teuthology.orchestra.console.smithi134:Power off 2024-03-25T19:41:10.827 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:41:10.850 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-03-25T19:41:10.851 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:10.865 INFO:teuthology.orchestra.console.smithi159:Power off 2024-03-25T19:41:10.866 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:41:10.875 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-25T19:41:10.887 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2024-03-25T19:41:10.887 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:10.909 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-25T19:41:14.876 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:14.910 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:14.986 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-25T19:41:15.020 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-25T19:41:18.991 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:19.021 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:19.102 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-25T19:41:19.130 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2024-03-25T19:41:19.230 INFO:teuthology.orchestra.console.smithi159:Power off completed 2024-03-25T19:41:23.106 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:23.216 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-03-25T19:41:23.317 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-03-25T19:41:23.421 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6319.620280742645 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ac3ddef8859e4e748788aada5fc51344 status: fail success: false 2024-03-25T19:41:23.422 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:41:23.511 INFO:teuthology.run:FAIL